How NNTP Works
Copy(c)right 2000 by GM Raymond

NEWSGROUPS: News you can Use

We need your help!

Last updated: 8/1/2000

By now almost everyone knows the Internet consist of two major areas of world services. E-Mail and WEB sites. However there is, and always has been, a third element called Unet or UseGroups. Today the latter is currently the least used or understood . Originally, it was a cornerstone of the newly evolving network we now call the Internet.

All three primary services are supported by infrastructure on the Internet Provider side of things. This means that your provider must operate and maintain three different categories of servers (originally called Daemon's) to offer these amenities. Remember, a server is nothing more than a PC very much like yours at home (maybe a bit more powerful) running special software.

Here is a brief rundown of the types of servers your ISP operates. DNS for resolving their verbose domains to his clients IP's. To provide e-mail, an ISP operates an SMTP server (simplified mail transfer protocol) which receives your outbound mail and relays it to your destination; a POP3 server is needed for mail arriving at your provider and addressed to you; an HTTP server to support customers web pages; and finally, an NNTP server to provide the newsgroups. (POP stands for Post Office Protocol; HTTP stands for Hyper Text Transfer Protocol and DNS for Domain Name Server). I have written other articles to cover e-mail and web services so lets now discuss Newsgroups.

NNTP stands for Network News Transfer Protocol, the underlying data packet system that makes it all work.

In the case of NNTP, the provider also depends on NNTP feed servers, typically operated by the Backbones or main thoroughfare's onto the Internet highway that are owned, operated and maintained by the major telecommunications companies in the world like AT&T, Sprint, MCI, etc etc.

A very simple analogy for a Newsgroups is to compare it to a cork bulletin boards on the wall of an office hallway. Anyone can stop, read, or pin up a note themselves if they are so inclined. Imagine a situation where 37,000 cork boards exist in ever building, each one specialized, or covering a very narrow subject topic, such as "persian. cats.buy.or.sell".

Also imagine couriers, who run around every day duplicating each unique cork board's content, at every location they appear.

NNTP can also be thought of as the final evolutionary step in the development and demise of the old Bulletin Board Services (or BBS's); although for several years the two things existed concurrently. Today there are very few major BBS's still in operation.

So much for the analogy, now here's how it really works.

Let's start with you wanting to post a message to "persian. cats.buy.or.sell". You type up your message, called an Article, and send it. It goes to your Providers NNTP server where it is immediately available to all the rest of their users. But it is also forwarded from your ISP to his NNTP news feed (typically at the backbone he buys net connectivity from). That hub in turn send it out to all other hubs who in turn send it out to all their subscribing Providers. As you can see, in no time at all, your article is literally available to all users worldwide.

Aside from sending new articles, you can also reply to existing articles. And when this happens, all replies are threaded to the original article to make it easy for readers to follow the continuity of all the replies.

When Articles are created, they consist of two component parts, the Header, which contains the posting date, author, subject and the line count of the article; and the Body, which contains the actual message or article.

A great time saving feature of the system is the ability to first download only the Headers (which are very short and fast to download) rather than the entire article. This gives the user quick access to all current subjects and the ability to scan for items of interest prior to wasting time downloading unwanted article bodies.

To read and post newsgroup articles, you will need a good Newsgroup client similar to how you need a good e- mail client to receive and send e-mail. Many Browsers have Newsreaders built into them. But if you want as much versatility as possible, you will want a stand alone client like FreeAgent, Gravity or any one of several others available on the Net.

The following information will be centered around the most popular News reader, Agent. However what you learn here can easily be applied to all other News Readers as they all follow the same basic modes of operation.

If you are starting from scratch, the very first thing you want to do is download a listing of all the Newsgroups handled by your ISP's NNTP server. Note: just because 37,000 groups exist worldwide does not necessarily mean your ISP will carry them all. There is a lot of local censoring by goody two shoes who like to think they know what is, and is not, good for you. Sometimes this can be a determining factor in picking a Provider. And while I am mentioning this, you should know that there are other options to using your ISP's NNTP server. You can pay additional money to cherry pick a bigger commercial NNTP service with zero restrictions. For that matter, there are even FREE NNTP operations supported by Universities and Public Institutions.

You can find a list of free or public NNTP servers Here

Ok, lets assume you have downloaded the list of 37,000 News groups. Thats a big list to go wadding through just out of curiosity. However, many people do. If your time is important, just use the "Find" search choice on your reader client to zero in on a few key works that will likely be contained in the newsgroups name. Once you locate groups with the word "coin" in them (as an example), the next task will be to mark those groups of interest for viewing and posting. This is know as Subscribing to a group; only in this situation, it implies no real economic or otherwise responsibility. The act of Subscribing is all internal to your reader and simply marks your groups of interest for joint viewing in a single and manageable window.

A good news reader will offer a lot of flexibility. Example, you can selectively download headers from just one, several, or all subscribed groups. Or you can do the same for headers "and" bodies (if you have a lot of free time).

And, for those crafty enough to know exactly what they are looking for, you can set up "Filters" that will watch the incoming headers for key words, such as the specific author, subject or date, and automatically MARK those articles for easy identification and body retrieval immediately or at a later time.

You also have a wide array of options available in how you delete old headers, or mark they as unread while also differentiating them from new headers you download the next session. The possibilities are only limited by your imagination and needs.

Next, is the subject of attachments. Attachments, just as with conventional e-mail, are a way to send out non text files (called binaries) on a text only system. Examples of binaries are graphic files, programs, sound files etc. If you are a vocalist, and wish to record a song in the new MP3 format, and offer it to the public domain, this is an excellent way to distribute it.

Whereas e-mail attachments all adhere to the base64 or MIME standard for encoding, Newsgroups, having been born on older UNIX computers, are still adhering to a somewhat less efficient system called uuencoding. Uuencoding converts an 8 bit binary format to a 7 bit text format so it can continue to be transmitted on all the older 7bit message servers still in use on the net.

Most versatile news readers like Agent will offer internal decoding and encoding for binary attachments. Agent has the older uuencode cypher along with the more contemporary base64 or MIME cypher. When you elect to attach a binary file to an article, the entire process of encoding or decoding is done in the background and rather automatically without any user intervention.

Agent Newsreader offers a lot of flexibility in how you can organize its desktop (main window or work area). My preference is to use all of the available desktop space for whatever task is at hand at the moment. When organized in this way, three big buttons and one screen length wise bar will do a great deal of the work.

The first three are labeled:

Groups\Folders: This works with the big long bar, and depending on how it is set, will show either "All Groups", "New Groups", "Subscribed Groups" or "Folders" in the main view area.

Message List: Shows the Headers in the main view area.

Messages: Shows the Bodies (Articles) in main view area.

The long horizontal bar below these three buttons is actually a toggle; meaning any of four views can be had depending on how many times you click the bar (up to four clicks) when it cycles back to the beginning again. The four views are: Subscribed Groups, New Groups, Folders and All Groups.

I have found very little use for the "Folders" function but their purpose is to store articles that you wish to keep. However, there is a much easier way to do this using the "Lock" article function.

Let's walk through a typical daily session. Agent opens showing the Groups you have already subscribed to (assuming thats how you left it - and you should have). You may have several or more listed but you are only interested in downloading Headers for two of them.

Mark the first with a single click. Hold Ctrl while you click and mark the next. If the ones you want marked are all together, click the first and while holding down the button, drag the cursor up to the last and release. Or, if you want them all marked, hit Ctrl- A ,or pick "Select All" under the Edit Menu up at the very top of the client window. Or, click the top group, hold down the Shift key and click the last. Say, have you noticed how many different ways there are to MARK items? Yes, its another marvel of Windows.

Now that your selected groups are marked, its time to start downloading the headers. You can go to the top Menu again a click on "On Line". Next you will click on "Get new headers in selected groups" and the download of headers begins. NOTE: you could also configure a Tool Bar Button to do that same job!

After the headers are all downloaded, a number will appear to the left of the Group name. This tells how many Articles are available. Technically, depending on how Agent is configured, it might be a lessor number that represents just major articles and not all the replies or threads of that article.

Now you click and mark (highlight) a Group that interest you. To see the actual headers, Click the "Message List" button. If you had a filter set, any header meeting the criterion of your filter will have a small blue Arrow appended to the immediate left of the header line.

To download and see the body of any specific header, mark it and double click. This action will both download the body and display it after it is received. After reading, you can toggle back to the Headers display by clicking on "Message List" again. Or, you could reply to the article you just read with a Follow Up article by selecting the "Post" menu choice from above.

When viewing headers, there are several Title bars at the top of the viewing window. They are toggles that permit changing how you view the content of the windows. For instance, if you click Subject, all headers will be re indexed with all identical subjects together. If you click Date, the headers will be re arranged by Date in descending order. If you click the Size so Threads shows, the display shows message groups that are bound by the same subject, along with a number indicating the quantity of threads under that same subject. Clicking Threads causes it to revert to "Size" and list all headers separately.

To post a new article from scratch, highlight the group and from the "Post" menu, select "New Usenet Message". After defining a subject and composing your article, click on the "Send" button far left at top of composing window.

If you downloaded an article that contained an attachment, the attachment will be placed into a predefined directory typically called something like c:\download. Agent also offer the option of "launching" the attachment which can be particularly useful if the attachment is a graphic image and you desire to view it immediately. Again, a special Launch button can be placed on the Tool Bar to make this job easy.

Finally, I want to end this section with a few words about filters. Agent filters are based on Boolean expressions. Lets examine several simple filters.

"coins AND old"

"coins OR old"

In the first example BOTH words must appear somewhere. In the second filter if either word appears the case is met.

Filters can get as complicated as your mind permits by compounding expressions; here is an example of a "simple" compound filter: "coins AND mint OR old".

There are a lot of other boolean expression that can be added to filters. Read Agent's Help for more details.

Now for a word about the 37,000 newsgroups themselves.

In the early days of the net, when it was mostly Universities and Government Research Centers exchanging information among students, Professors and Scientist, a very strict kind of netiquette was adhered too. Meaning if a group was formed titled nuclear.fusion, it was a sacrilege to bring up any other subject, especially anything that might be construed as personal. People who violated this rule were often publicly berated or "flamed" as the lingo now implies.

It was during these early days that many of the current abbreviations and emoicons were created as a way of embracing polite brevity and also expressing some kind of universally understood human emotion. Here is an example of poor form. TO TYPE IN ALL CAPITAL LETTERS is considered rude and the equivalent of shouting at someone.

The number of Emoicons in use today is mind boggling so I will only cover a very few here for purposes of illustration.

<G> big grin
<g> little grin
<S> smile <IMHO> in my humble opinion
<OIC> oh I see
<RTFM> read the frekin manual
<ROFL> rolling on the floor laughing
<FUBAR> futzed up beyond any repair
<NBD> no big deal
:-) a smile
;-) a cynical smile
:-( a frown

Today, with the explosive growth of the Net, and the emergence onto the Net of the average Joe, a new set of newsgroups was required. The ALTternative or alt. news groups. Today the ALT is the largest of any types of groups.

All newsgroups are administered and approved by a division of the Internic called UuNet. All application for new groups must be made through a very strict application process and enough preliminary user support must actually be demonstrated to get in the door. Meaning if you wish to start another public group, you had better have a lot of friends ready to start an e- mail campaign on your behalf.

Groups are generally non moderated, meaning anything you post goes right through. Moderated groups either have a watchdog to chastise those who post out of subject articles; or, someone who must approve every post before allowing it to go public on their group. Many groups publish an FAQ (frequently asked questions) giving al the pertinent details on what their group is all about. Generally the FAQ is always available for download off the group.

There are also Private newsgroups, which are generally only carried on the local ISP's NNTP server. Factually, beyond the hubs, no private ISP is required by any rules to carry everything a hub supplies.

Additionally, aging or how long old articles remain on a particular server is not a rule set in stone either. The number of days aging articles are allowed to remain on a server is up to the System Administrator of that service but is typically 30 days.

Delete commands can be issued by a newsreader to remove articles YOU posted and subsequently be removed from the entire newsworld; but, most servers are set to ignore the Delete command to prevent non authors and hackers from destroying articles out of maliciousness.

Some free commercial NNTP services who profit from advertising, claim to archive news articles up to several years and make powerful search engines available to locate specific articles of interest based on key words.

One such service is DowJonesNews

Like e-mail, news articles may contain LINKS which, depending on your news reader, allow you to immediately jump to a web site vi your default web browser or send an e-mail vi your default e-mail program.



Return to Home Page!