Package org.htmlparser.beans
Class StringBean
java.lang.Object
org.htmlparser.visitors.NodeVisitor
org.htmlparser.beans.StringBean
- All Implemented Interfaces:
Serializable
Extract strings from a URL.
Text within <SCRIPT></SCRIPT> tags is removed.
The text within <PRE></PRE> tags is not altered.
The property Strings
, which is the output property is null
until a URL is set. So a typical usage is:
StringBean sb = new StringBean (); sb.setLinks (false); sb.setReplaceNonBreakingSpaces (true); sb.setCollapse (true); sb.setURL ("http://www.netbeans.org"); // the HTTP is performed here String s = sb.getStrings ();You can also use the StringBean as a NodeVisitor on your own parser, in which case you have to refetch your page if you change one of the properties because it resets the Strings property:
StringBean sb = new StringBean (); Parser parser = new Parser ("http://cbc.ca"); parser.visitAllNodesWith (sb); String s = sb.getStrings (); sb.setLinks (true); parser.reset (); parser.visitAllNodesWith (sb); String sl = sb.getStrings ();According to Nick Burch, who contributed the patch, this is handy if you don't want StringBean to wander off and get the content itself, either because you already have it, it's not on a website etc.
- See Also:
-
Field Summary
FieldsModifier and TypeFieldDescriptionprotected StringBuffer
The buffer text is stored in while traversing the HTML.protected boolean
Iftrue
sequences of whitespace characters are replaced with a single space character.protected int
The state of the collapse processiung state machine.protected boolean
Settrue
when traversing a PRE tag.protected boolean
Settrue
when traversing a SCRIPT tag.protected boolean
Settrue
when traversing a STYLE tag.protected boolean
Iftrue
the link URLs are embedded in the text output.protected Parser
The parser used to extract strings.protected PropertyChangeSupport
Bound property support.protected boolean
Iftrue
regular space characters are substituted for non-breaking spaces in the text output.protected String
The strings extracted from the URL.static final String
Property name in event where the 'collapse whitespace' state changes.static final String
Property name in event where the connection changes.static final String
Property name in event where the 'embed links' state changes.static final String
Property name in event where the 'replace non-breaking spaces' state changes.static final String
Property name in event where the URL contents changes.static final String
Property name in event where the URL changes. -
Constructor Summary
Constructors -
Method Summary
Modifier and TypeMethodDescriptionvoid
Add a PropertyChangeListener to the listener list.protected void
Appends a newline to the buffer if there isn't one there already.protected void
collapse
(StringBuffer buffer, String string) Add the given text collapsing whitespace.protected String
Extract the text from a page.boolean
Get the current 'collapse whitespace' state.Get the current connection.boolean
getLinks()
Get the current 'include links' state.boolean
Get the current 'replace non breaking spaces' state.Return the textual contents of the URL.getURL()
Get the current URL.static void
Unit test.void
Remove a PropertyChangeListener from the listener list.void
setCollapse
(boolean collapse) Set the current 'collapse whitespace' state.void
setConnection
(URLConnection connection) Set the parser's connection.void
setLinks
(boolean links) Set the 'include links' state.void
setReplaceNonBreakingSpaces
(boolean replace) Set the 'replace non breaking spaces' state.protected void
Fetch the URL contents.void
Set the URL to extract strings from.protected void
updateStrings
(String strings) Assign theStrings
property, firing the property change.void
visitEndTag
(Tag tag) Resets the state of the PRE and SCRIPT flags.void
visitStringNode
(Text string) Appends the text to the output.void
Appends a NEWLINE to the output if the tag breaks flow, and possibly sets the state of the PRE and SCRIPT flags.Methods inherited from class org.htmlparser.visitors.NodeVisitor
beginParsing, finishedParsing, shouldRecurseChildren, shouldRecurseSelf, visitRemarkNode
-
Field Details
-
PROP_STRINGS_PROPERTY
Property name in event where the URL contents changes.- See Also:
-
PROP_LINKS_PROPERTY
Property name in event where the 'embed links' state changes.- See Also:
-
PROP_URL_PROPERTY
Property name in event where the URL changes.- See Also:
-
PROP_REPLACE_SPACE_PROPERTY
Property name in event where the 'replace non-breaking spaces' state changes.- See Also:
-
PROP_COLLAPSE_PROPERTY
Property name in event where the 'collapse whitespace' state changes.- See Also:
-
PROP_CONNECTION_PROPERTY
Property name in event where the connection changes.- See Also:
-
mPropertySupport
Bound property support. -
mParser
The parser used to extract strings. -
mStrings
The strings extracted from the URL. -
mLinks
protected boolean mLinksIftrue
the link URLs are embedded in the text output. -
mReplaceSpace
protected boolean mReplaceSpaceIftrue
regular space characters are substituted for non-breaking spaces in the text output. -
mCollapse
protected boolean mCollapseIftrue
sequences of whitespace characters are replaced with a single space character. -
mCollapseState
protected int mCollapseStateThe state of the collapse processiung state machine. -
mBuffer
The buffer text is stored in while traversing the HTML. -
mIsScript
protected boolean mIsScriptSettrue
when traversing a SCRIPT tag. -
mIsPre
protected boolean mIsPreSettrue
when traversing a PRE tag. -
mIsStyle
protected boolean mIsStyleSettrue
when traversing a STYLE tag.
-
-
Constructor Details
-
StringBean
public StringBean()Create a StringBean object. Default property values are set to 'do the right thing':Links
is setfalse
so text appears like a browser would display it, albeit without the colour or underline clues normally associated with a link.ReplaceNonBreakingSpaces
is settrue
, so that printing the text works, but the extra information regarding these formatting marks is available if you set it false.Collapse
is settrue
, so text appears compact like a browser would display it.
-
-
Method Details
-
carriageReturn
protected void carriageReturn()Appends a newline to the buffer if there isn't one there already. Except if the buffer is empty. -
collapse
Add the given text collapsing whitespace. Use a little finite state machine:state 0: whitepace was last emitted character state 1: in whitespace state 2: in word A whitespace character moves us to state 1 and any other character moves us to state 2, except that state 0 stays in state 0 until a non-whitespace and going from whitespace to word we emit a space before the character: input: whitespace other-character state\next 0 0 2 1 1 space then 2 2 1 2
- Parameters:
buffer
- The buffer to append to.string
- The string to append.
-
extractStrings
Extract the text from a page.- Returns:
- The textual contents of the page.
- Throws:
ParserException
- If a parse error occurs.
-
updateStrings
Assign theStrings
property, firing the property change.- Parameters:
strings
- The new value of theStrings
property.
-
setStrings
protected void setStrings()Fetch the URL contents. Only do work if there is a valid parser with it's URL set. -
addPropertyChangeListener
Add a PropertyChangeListener to the listener list. The listener is registered for all properties.- Parameters:
listener
- The PropertyChangeListener to be added.
-
removePropertyChangeListener
Remove a PropertyChangeListener from the listener list. This removes a registered PropertyChangeListener.- Parameters:
listener
- The PropertyChangeListener to be removed.
-
getStrings
Return the textual contents of the URL. This is the primary output of the bean.- Returns:
- The user visible (what would be seen in a browser) text.
-
getLinks
public boolean getLinks()Get the current 'include links' state.- Returns:
true
if link text is included in the text extracted from the URL,false
otherwise.
-
setLinks
public void setLinks(boolean links) Set the 'include links' state. If the setting is changed after the URL has been set, the text from the URL will be reacquired, which is possibly expensive.- Parameters:
links
- Usetrue
if link text is to be included in the text extracted from the URL,false
otherwise.
-
getURL
Get the current URL.- Returns:
- The URL from which text has been extracted, or
null
if this property has not been set yet.
-
setURL
Set the URL to extract strings from. The text from the URL will be fetched, which may be expensive, so this property should be set last.- Parameters:
url
- The URL that text should be fetched from.
-
getReplaceNonBreakingSpaces
public boolean getReplaceNonBreakingSpaces()Get the current 'replace non breaking spaces' state.- Returns:
true
if non-breaking spaces (character '\u00a0', numeric character reference   or character entity reference ) are to be replaced with normal spaces (character '\u0020').
-
setReplaceNonBreakingSpaces
public void setReplaceNonBreakingSpaces(boolean replace) Set the 'replace non breaking spaces' state. If the setting is changed after the URL has been set, the text from the URL will be reacquired, which is possibly expensive.- Parameters:
replace
-true
if non-breaking spaces (character '\u00a0', numeric character reference   or character entity reference ) are to be replaced with normal spaces (character '\u0020').
-
getCollapse
public boolean getCollapse()Get the current 'collapse whitespace' state. If set totrue
this emulates the operation of browsers in interpretting text whereuser agents should collapse input white space sequences when producing output inter-word space
. See HTML specification section 9.1 White space http://www.w3.org/TR/html4/struct/text.html#h-9.1.- Returns:
true
if sequences of whitespace (space '\u0020', tab '\u0009', form feed '\u000C', zero-width space '\u200B', carriage-return '\r' and NEWLINE '\n') are to be replaced with a single space.
-
setCollapse
public void setCollapse(boolean collapse) Set the current 'collapse whitespace' state. If the setting is changed after the URL has been set, the text from the URL will be reacquired, which is possibly expensive. The internal state of the collapse state machine can be reset with code like this:setCollapse (getCollapse ());
- Parameters:
collapse
- Iftrue
, sequences of whitespace will be reduced to a single space.
-
getConnection
Get the current connection.- Returns:
- The connection that the parser has or
null
if it hasn't been set or the parser hasn't been constructed yet.
-
setConnection
Set the parser's connection. The text from the URL will be fetched, which may be expensive, so this property should be set last.- Parameters:
connection
- New value of property Connection.
-
visitStringNode
Appends the text to the output.- Overrides:
visitStringNode
in classNodeVisitor
- Parameters:
string
- The text node.
-
visitTag
Appends a NEWLINE to the output if the tag breaks flow, and possibly sets the state of the PRE and SCRIPT flags.- Overrides:
visitTag
in classNodeVisitor
- Parameters:
tag
- The tag to examine.
-
visitEndTag
Resets the state of the PRE and SCRIPT flags.- Overrides:
visitEndTag
in classNodeVisitor
- Parameters:
tag
- The end tag to process.
-
main
Unit test.- Parameters:
args
- Pass arg[0] as the URL to process.
-