This Fall, I celebrate five years of blogging. I have written tons of web development stuff at Perishable Press, lots of helpful WordPress stuff at Digging into WordPress, some philosophical stuff at mindfeed.org, creative/artistic stuff at Dead Letter Art, jQuery stuff at jQuery Mix, and some business-related web-design stuff at Monzilla Media. Plus a bunch of interviews, guest posts, and other blogging projects. So yeah, lots of blogging and writing during the past five years. And they just flew by.

Despite what the haters may say, there are some tangible benefits to blogging. As I write, I continue to learn a great deal – not just about the fine art of writing, but also about the nature of the audience, social media, and the Web in general. There’s a lot to it, more than you may realize. Looking back during my recent hiatus, I enjoyed the opportunity to reflect on the past and contemplate lessons learned, future goals, and what it all means. Here are some of my thoughts, strategies, and lessons learned after five years of blogging..

Original Goals and Strategy

Getting into the blogging game around five years ago, I really had no idea what I was doing. Back then, there were a few “pro” bloggers that paved the way for a lot of folks (mostly big-money bloggers et al), but for the most part the frontier was wide open. When I first jumped in, it was because I was completely smitten by the web-design/WordPress/blogging bug. I loved web design, and found blogging about my experiences quite rewarding. As I first delved into the online game, my goals were rather simple:

• Get in the game and build-up and establish my first blogging site, Perishable Press
• Share as much knowledge as possible about creativity, graphic design, web design, etc.
• Become a better designer in the process of writing about and doing web design/development

Now keep in mind that these were intuitive goals that just seemed like the right thing to do. Most of the content I read on the Web back then was focused strictly on web-design, web-standards, and specific programming langauges – not a lot of stuff on how to blog, how to make money, how to be a rockstar, and all of that BS. A lot of that crap didn’t really begin to hit the scene hard until a few years later, and by that time I knew well-enough that it wasn’t how I wanted to do things. I wanted to do it right from the beginning: build a strong foundation, help as many people as possible, and continue to improve my skillz. All of that “make-money-blogging” and “be-a-rockstar” crap was just too superficial and pathetic. So, to generalize my initial strategy for working on the Web, here it is:

• Work hard
• Learn much
• Share much

Anything less than that and I would be cheating myself from really getting the most out of the experience. I think this strategy is ideal for any activity, whether online or off. Unfortunately, the mainstream is not interested in any of these practices, unless you count hard work motivated by greed. So you’ve got 90% of the online game doing the exact opposite:

• Avoid work
• Avoid learning
• Share?! Are you kidding me?!

Fortunately, hard work, education, and generosity always pay off. It may take longer than cheating, lying, and stealing, but the rewards are infinitely more beneficial and rewarding. It can get frustrating, however, watching complete and utter sellouts flying past you on their way to the top, but once the fame and fortune is gone, they have nothing. Meanwhile, you gain the experience, education, and wisdom that will enrich your life long after your time on the Web.

Lessons Learned

Perhaps the most profound thing I have learned while working on the Web is that virtual social skills definitely translate into tangible, real-world social skills, despite what they may tell you. Ask any seasoned blogger: the key to social success is maintaining diplomacy and understanding in all situations:

• Responding to comments
• Dealing with attacks
• Engaging other blogs
• Providing feedback

You’re gonna get some wicked comments, evil people who have nothing better to do than troll your site and try to bring you down. You learn quickly the best way to deal with trolls and diffuse potentially degrading situations is by maintaining a level head. Keep your cool and it’s possible to turn even the biggest haterz into complete fanz. The same principle applies in the real world, where there are just as many if not more haterz and villainz to deal with. Thankfully, the years spent dealing with people on the Web have helped me understand how to deal with them in “real-life” – people are people whether online or off. When dealing with an irate neighbor, I just slip it into “comment-response” mode and turn the situation around.

You will never find a more wretched hive

Despite my best efforts at maintaining a positive vibe, it is hard to ignore all of the wretchedness and wickedness on the Web these days. There are waay too many scumbags who couldn’t care less about anyone else. Most of the time, these lower lifeforms manifest as relentless spam, scraped content, and adsense profits. The Web is flooded with shallow, mindless worms who just want the money. Sadly:

• There is too much empty, shallow, echoed, meaningless, pointless, worthless content
• There are waay too many ads, and 90% of them are completely transparent and awful
• There is waay too much noise – getting a good signal is becoming exceedingly difficult
• There is too much misinformation, deception, ignorance, intolerance, and misinformation
• There is too much ego – apparently, it’s all about hyping up the latest loser
• There is too much selfishness – whatever happened to linking out to other sites?

Given that, it’s a miracle that any decent, sharing, honest people would have any interest in the Web at all. I mean, if you are a decent person and just starting out on the Web, there is a lot working against you. 90% of the people on the Web won’t blink twice before stealing from you, sticking you with a fee, or spamming you to death. Seriously, if 90% of the people on the Web want you to fail, why even bother? I’ll tell you why: because of the other 10% of people who are honest, hard-working, decent folks making the Web a better place. I love to meet and work with the good guys – they indeed make it all worthwhile and enjoyable.

If there is a bright side to the Web

Not all of my “lessons learned” are focused on the negative. Despite what you just read, there are plenty of positives involved with blogging, social media, and online work in general. The main thing you have to keep in mind is that hard work pays off. Always has, always will. And the Web is no exception. Now, I’m not saying “build it and they will come,” but if you are determined, persistent, and motivated enough, success will be yours.

Another old saying is that “it’s not what you know, but who you know.” As much as I absolutely hate this notion, I cannot avoid the inescapable fact that it’s true. Take the time to network with your peers, your fanz, your family, friends, and anyone else worthwhile. It really is important. I have seen people go from zero to rockstar in less than a year by networking with the popular folks. Just do your best to keep it real – nobody likes to be used.

Success is not a linear progression

The further you go, the harder it gets. When you first start out, everything is wiide open, fresh and new. Possibilities are virtually endless, and there is nothing to lose. Going forward, you are shaped by your successes and failures. You do more of what clicks and less of what doesn’t. As you continue doing what works, you take fewer chances, limiting your possibilities and locking yourself into “safe” ways of thinking and doing. To make things fit with what works, your goals will change. For example, I started off writing at Perishable Press about anything and everything that had anything at all to do with creativity: art, photography, painting, web design, and so on. After little response to the artsy-fartsy stuff and a huge response to the web-design and WordPress stuff, I began focusing more and more on, well, web design and WordPress. Today, I am largely defined through that particular lens.

Popularity is a double-edged sword. When you first start out, you can blog about whatever you want because you know that nobody is listening anyway. You still blog your best blog in hopes of attracting attention, but ultimately you can feel free to write stuff that sucks. As you earn a following online, expectations keep you blogging for your audience. If you fail to provide what’s expected, chances are high that your readers will go elsewhere to get it. For example, let’s say I write about CSS and gain a huge following. Expectations keep me writing about CSS, because if I stray too far from it, *poof* – there goes my audience.

Also as you move up the ladder of success and popularity, you’re going to find that the competition gets extremely fierce. This is what I mean when I say that “success is not a linear progression.” It’s more like logarithmic or exponential or something, especially where competition is concerned. Think of success and popularity like a hill. There is tons of room at the bottom, where the circumference is largest. But as you climb, the hill gets increasingly smaller all the way to the top, where “there can be only one!” Naturally, along the way, as you climb along with everyone else, you’re going to meet bloggers and rockstars that you once admired climbing up with you. You may pass some of your peers, and discover that others are complete assholes. Because, you know, they are already halfway up the hill – it’s their spot, their success, their ego: “look how far I made it up the hill! Look how popular I am! Look how amazing and ..” You get the idea. Ego is a trap that will keep you from climbing further. When I meet someone with a giant ego, I love to feed it while passing along – it works to my advantage.

The Pros and cons of having an audience

Ultimately, when it comes to blogging, your audience defines you. There are pros and cons to having an audience of your own. Without an audience, you can do and say pretty much whatever you want with no issues. The larger your audience becomes, the more scrutiny your words are going to receive. This is both good and bad: you want people listening to what you have to say, but there will be more drama if you say something disagreeable. You also have much more to lose with a larger audience. A good name is more desirable than gold, so if you screw up and say something stupid, you may lose whatever reputation you have managed to build. The smaller your audience, the less you have to lose, and vice-versa.

Also, the more widely known you are for doing a specific thing, the more requests you’re going to get for help. Again, this can be both a blessing and a curse, depending on what it is that you do and your reasons for doing it. Web development is a perfect example. I get tons of emails asking for help with HTAccess, PHP, and JavaScript, but 90% of the time it is clearly implied that free help is requested. I don’t mind helping people for free when I have the time, but it would be nice to be valued. If someone asks for help and isn’t willing to compensate for it, they’re basically telling you that your service is not valued and not worth paying for. If you’re on your way up, be prepared for beggars, freeloaders, and leeches.

Perhaps the biggest problem most bloggers run into is trying to please everyone. Don’t even try – it’s impossible to do. No matter how hard you try, there will always be somebody that has a problem with what you are saying. Most of the time you’ll find 30% of your audience agrees with you, 30% disagrees with you, 30% doesn’t care either way, and the other 10% insists on freaking out and making a scene. Don’t feed the trolls, as they say. The key is to embrace that 30% of readers that actually “gets” you. Write for yourself first and them next. Along the way, keep in mind that people change and move on.

The Mainstream vs. Your Stream

We each have our own stream of consciousness, activity, and so on. On the web, as in life, certain topics are more popular than others. When you have a popular topic such as Megan Fox, you are getting into the mainstream. The more people like a particular subject, the more mainstream it’s going to be. Thus, if you are blogging about “making money”, it’s going to be much easier gaining a large following than if you were to blog about, say, differential equations. The more your blogging interests coincide with popular, mainstream topics, the easier it is to be popular.

That doesn’t mean you should run out and jump on the “Twighlight” bandwagon just to be popular. In fact, it’s just the opposite. The mainstream is where the numbers are, true, but it’s also where the least-common denominator resides. The mainstream is where the sheep swim. In my experience, the mainstream is the most dumbed-down, uninteresting, lowest-value content available. It’s there for one reason and one reason only: to make money by giving the masses what they want, which is typically entertainment, sex, drugs, and violence. You know exactly what I’m talking about here. It’s the reason Hollywood continues to churn out such pathetic garbage – because it sells.

Contrast the mainstream with your own stream. How much overlap is there? Using myself as an example, I see that certain interests of mine are very hot within the web-design community, at least for the moment. I like what most all web designers like: jQuery, CSS, and WordPress. But I also like a lot of other, less-popular things, like HTAccess, site security, and error logging. If I wanted to rise to the top, I could sell out and just write articles about CSS and jQuery, maybe throw down a few million top-10 lists, give away some free stuff and watch the traffic surge. But that’s not what I am all about. I like writing about esoteric topics, even if that means a smaller audience and less popular blog.

When I come to your site and see a million advertisements, a sidebar full of social-media crap, and a post containing a few weak-ass paragraphs about something that’s already been blogged about to death, a little part of me dies. Don’t be like that. Get a freakin’ clue and try a little harder not to be such an absolute sellout media whore. That’s what everyone else is doing – that’s what the mainstream is doing. And the mainstream sucks.

Success, prosperity, and satisfaction is possible by doing your own thing, swimming your own stream, being yourself. Know who you are, know what you like, be yourself, and share your experience. It’s better to enjoy a small audience that likes your stuff than to cater to a large audience with mainstream crap.

Assuming you achieve your goals, what would you rather have: a huge audience of pathetic, mainstream dittoheads or a smaller audience that actually shares similar interests and listens to what you have to say. For me, the answer is obvious.

The Narrow Way

To wrap things up, let me summarize my lessons learned after five years of blogging and working on the Web:

• Be yourself
• Be honest – no hype
• Be real – no fluff
• Be sincere, genuine, unique
• Don’t be lazy, selfish
• Think for yourself
• Take advantage of criticism
• Control temper, be patient
• Work hard, learn much, share often
• Be yourself

It’s a narrow path, but for me, it’s the only way to go.

Source: Perishable Press

Take your WordPress skills to the next level with Digging into WordPress!

Related articles

• A Few Steps Back
• Riding the Wave
• An Inside Look at the Hectic Schedule of an Internet Addict
• The Pros and Cons of Blogging
• Open Call for Guest Posts at Perishable Press
• A Social Networking Nightmare
• Fully Valid, SEO-Friendly Social Media Links for WordPress

Read more: http://perishablepress.com/press/2010/08/30/lessons-learned-after-5-years-of-blogging/

The 2010 User-Agent Blacklist blocks hundreds of bad bots while ensuring open-access for the major search engines: Google, Bing, Ask, Yahoo, et al. Blocking bad user-agents is an effective addition to any security strategy. It works like this: your site is getting hammered by rogue bots that waste valuable server resources and bandwidth. So you grab a copy of the 2010 UA Blacklist from Perishable Press, include it in your site’s root .htaccess file, and enjoy a more secure and better performing website. It’s that easy.

Proven Security

The 2010 UA Blacklist has been carefully constructed based on rigorous server-log analyses. Obsessive daily log monitoring reveals bad bots scanning for exploits, spamming resources, and wasting bandwidth. While analyzing malicious behavior, evil bots are identified and added to the UA Blacklist. Blocked user-agents are denied access to your site, increasing efficiency and providing safety for your visitors.

Better Performance, Better SEO

Search engines such as Google are placing more weight on speedy, fast-loading websites. If your site is plagued with resource-devouring, bandwidth-wasting bots, it’s performance is probably not as good as it should be. Even if your site looks fine on the surface, without proper protection bad bots can gobble your bandwidth and leech your server resources. A single malicious bot can make hundreds and thousands of requests in a very short period of time while scanning and probing for vulnerabilities. If Google visits while bad bots are hitting your site, your site’s SEO could suffer. Fortunately, the 2010 UA Blacklist protects your site against hundreds of nefarious bots, thereby fostering maximum performance for the search engines.

2010 User-Agent Blacklist

Here it is, presented as two sets of HTAccess directives:

RewriteCond %{HTTP_USER_AGENT} .*(Firs|exac|Cloak|Detect|uchoo|beaut|ASPSeek|swish|ICS\)|MSIE\ 6\.0\;\ Windows\ NT\;\ DigExt\)|pt\-BR\;\ rv\:1\.9\.0\.3\)\ Firefox\/3\.0|pt\-BR\;\ rv\:1\.9\.0\.18\)\ Firefox\/3\.0|\!susie|\$x0e|\%0a|\%0d|\@\$x|\_irc|\_works|\+select\+|\+union\+|\<\?|1\,\1\,1\,|3gse|4all|4anything|5\.1\;\ xv6875\)|59\.64\.153\.|85\.17\.|88\.0\.106\.|98|a\_browser|a1\ site|abac|abach|abby|aberja|abilon|abont|abot|accept|access|accoo|accoon|aceftp|acme|active|address|adopt|adress|advisor|agent|ahead|aihit|aipbot|alarm|albert|alek|alexa\ toolbar\;\ \(r1\ 1\.5\)|alltop|alma|alot|alpha|america\ online\ browser\ 1\.1|amfi|amfibi|anal|andit|anon|ansearch|answer|answerbus|answerchase|antivirx|apollo|appie|arach|archive|arian|aboutoil|asps|aster|atari|atlocal|atom|atrax|atrop|attrib|autoh|autohot|av\ fetch|avsearch|axod|axon|baboom|baby|back|baid|bali|bandit|barry|basichttp|batch|bdfetch|beat|become|bee|beij|betabot|biglotron|bilgi|bison|bitacle|bitly|blaiz|blitz|blogl|blogscope|blogzice|bloob|blow|bord|boi|bond|boris|bost|bot\.ara|botje|botw|bpimage|brand|brok|broth|browseabit|browsex|bruin|bsalsa|bsdseek|built|bulls|bumble|bunny|busca|busi|buy|bwh3|cafek|cafi|camel|cand|captu|casper|catch|ccbot|ccubee|cd34|ceg|cfnetwork|cgichk|cha0s|chang|chaos|char|char\(|chase\ x|check\_http|checker|checkonly|chek|chill|chttpclient|cipinet|cisco|cita|citeseer|clam|claria|claw|clush|coast|code\.com|cogent|coldfusion|coll|collect|comb|combine|commentreader|common|compan|compatible\-|conc|conduc|contact|control|contype|conv|cool|copi|copy|coral|corn|cosmos|costa|cowbot|cr4nk|craft|cralwer|crank|crap|crawler0|crazy|cres|cs\-cz|cshttp|cuill|CURI|curl|curry|custo|cute|cyber|cz3|czx|daily|dalvik|daobot|dark|darwin|data|daten|dcbot|dcs|dds\ explorer|deep|deps|detect|dex|diam|diibot|dillo|ding|disc|disp|ditto|dlc|doco|dotbot|drag|drec|dsdl|dsok|dts|duck|dumb|eag|earn|earthcom|easydl|ebin|echo|edco|egoto|elnsb5|email|emer|empas|encyclo|enfi|enhan|enterprise\_search|envolk|erck|erocr|eventax|evere|evil|ewh|exploit|expre|extra|eyen|fang|fast|fastbug|faxo|fdse|feed24|feeddisc|feedhub|fetch|filan|fileboo|fimap|find|firebat|firedownload\/1\.2pre\ firefox\/3\.6|firefox\/0|firefox\/1|firefox\/2|firefox\/3\.0|firefox\/3\.0\.10|firs|flam|flash|flexum|flip|fly|focus|fooky|forum|forv|fost|foto|foun|fount|foxy\/1\;|free|friend|frontpage|fuck|fuer|futile|fyber|gais|galbot|gbpl|gecko\/2001|gecko\/2002|gecko\/2006|gecko\/2009042316|gener|geni|geo|geona|geth|getr|getw|ggl|gira|gluc|gnome|go\!zilla|goforit|goldfire|gonzo|google\ wireless|googlebot\-image|gosearch|got\-it|gozilla|grab|graf|greg|grub|grup|gsa\-cra|gsearch|gt\:\:www|guidebot|guruji|gyps|haha|hailo|harv|hash|hatena|hax|head|helm|herit|heritrix|hgre|hippo|hloader|hmse|hmview|holm|holy|hotbar\ 4\.4\.5\.0|hpprint|httpclient|httpconnect|httplib|human|huron|hverify|hybrid|hyper|iaskspi|ibm\ evv|iccra|ichiro|icopy|ida|ie\/5\.0|ieauto|iempt|iexplore\.exe|ilium|ilse|iltrov|indexer|indy|ineturl|infonav|innerpr|inspect|insuran|intellig|interget|internet\_explorer|internet\x|intraf|ip2|ipsel|irlbot|isc\_sys|isilo|isrccrawler|isspi|jady|jaka|jam|jenn|jet|jiro|jobo|joc|jupit|just|jyx|jyxo|kash|kazo|kbee|kenjin|kernel|keywo|kfsw|kkma|kmc|know|kosmix|krae|krug|ksibot|ktxn|kum|labs|lanshan|lapo|larbin|leech|lets|lexi|lexxe|libby|libcrawl|libcurl|libfetch|libweb|libwww|liferea|light|linc|lingue|linkcheck|linklint|linkman|lint|list|litefeeds|livedoor|livejournal|liveup|lmq|locu|london|lone|loop|lork|lth\_|lwp|mac\_f|magi|magp|mail\.ru|main|majest|mam|mama|mana|marketwire|masc|mass|mata|mvi|mcbot|mecha|mechanize|mediapartners|metadata|metalogger|metaspin|metauri|mete|mib\/2\.2|microsoft\.url|microsoft\_internet\_explorer|mido|miggi|miix|mindjet|mindman|mips|mira|mire|miss|mist|mizz|mj12|mlbot|mlm|mnog|moge|moje|mooz|more|mouse|mozdex) [NC]
RewriteRule ^.*$ - [G]

RewriteCond %{HTTP_USER_AGENT} .*(Windows\ NT\ 6\.1\;\ tr\;\ rv\:1\.9\.2\.6\)|mozilla\/0|mozilla\/1|mozilla\/2|mozilla\/3|mozilla\/4\.61\ \[en\]|mozilla\/firefox|mpf|msie\ 1|msie\ 2|msie\ 3|msie\ 4|msie\ 5|msie\ 6\.0\-|msie\ 6\.0b|msie\ 7\.0a1\;|msie\ 7\.0b\;|msie6xpv1|msiecrawler|msnbot\-media|msnbot\-products|msnptc|msproxy|msrbot|musc|mvac|mwm|my\_age|myapp|mydog|myeng|myie2|mysearch|myurl|nag|name|naver|navr|near|netants|netcach|netcrawl|netfront|netinfo|netmech|netsp|netx|netz|neural|neut|newsbreak|newsgatorinbox|newsrob|newt|next|ng\-s|ng\/2|nice|nikto|nimb|ninja|ninte|nog|noko|nomad|norb|note|npbot|nuse|nutch|nutex|nwsp|obje|ocel|octo|odi3|oegp|offby|offline|omea|omg|omhttp|onfo|onyx|openf|openssl|openu|opera\ 2|opera\ 3|opera\ 4|opera\ 5|opera\ 6|opera\ 7|orac|orbit|oreg|osis|our|outf|owl|p3p\_|page2rss|pagefet|pansci|parser|patw|pavu|pb2pb|pcbrow|pear|peer|pepe|perfect|perl|petit|phoenix\/0\.|php|phras|picalo|piff|pig|pingd|pipe|pirs|plag|planet|plant|platform|playstation|plesk|pluck|plukkie|poe\-com|poirot|pomp|post|postrank|powerset|preload|press|privoxy|probe|program\_shareware|protect|protocol|prowl|proxie|proxy|psbot|pubsub|puf|pulse|punit|purebot|purity|pyq|pyth|query|quest|qweer|radian|rambler|ramp|rapid|rawdog|rawgrunt|reap|reeder|refresh|reget|relevare|repo|requ|request|rese|retrieve|rip|rix|rma|roboz|rocket|rogue|rpt\-http|rsscache|ruby|ruff|rufus|rv\:0\.9\.7\)|salt|sample|sauger|savvy|sbcyds|sbider|sblog|sbp|scagent|scanner|scej\_|sched|schizo|schlong|schmo|scorp|scott|scout|scrawl|screen|screenshot|script|seamonkey\/1\.5a|search17|searchbot|searchme|sega|semto|sensis|seop|seopro|sept|sezn|seznam|share|sharp|shaz|shell|shelo|sherl|shim|shopwiki|silurian|simple|simplepie|siph|sitekiosk|sitescan|sitevigil|sitex|skam|skimp|sledink|sleip|slide|sly|smag|smurf|snag|snapbot|snapshot|snif|snip|snoop|sock|socsci|sogou|sohu|solr|some|soso|spad|span|spbot|speed|sphere|spin|sproose|spurl|sputnik|spyder|squi|sqwid|sqworm|ssm\_ag|stack|stamp|statbot|state|steel|stilo|strateg|stress|strip|style|subot|such|suck|sume|sunos\ 5\.7|sunrise|superbot|superbro|supervi|surf4me|surfbot|survey|susi|suza|suzu|sweep|sygol|synapse|sync2it|systems|szukacz|tagger|tagoo|tagyu|take|talkro|tamu|tandem|tarantula|tbot|tcf|tcs\/1|teamsoft|tecomi|teesoft|teleport|telesoft|tencent|terrawiz|test|texnut|thomas|tiehttp|timebot|timely|tipp|tiscali|titan|tmcrawler|tmhtload|tocrawl|todobr|tongco|toolbar\;\ \(r1|topic|topyx|torrent|track|translate|traveler|treeview|tricus|trivia|trivial|true|tunnel|turing|turnitin|tutorgig|twat|tweak|twice|tygo|ubee|ultraseek|unavail|unf|universal|unknown|upg1|uptime|urlbase|urllib|urly|user\-agent\:|useragent|usyd|vagabo|valet|valid|vamp|vci|veri\~li|verif|versus|via|virtual|visual|void|voyager|vsyn|w0000t|w3c|w3m|w3search|walhello|walker|wand|waol|watch|wavefire|wbdbot|weather|web\.ima|web2mal|webarchive|webbot|webcat|webcor|webcorp|webcrawl|webdat|webdup|webgo|webind|webis|webitpr|weblea|webmin|webmoney|webp|webql|webrobot|webster|websurf|webtre|webvac|webzip|wells|wep\_s|wget|whiz|widow|win67|windows\-rss|windows\ 2000|windows\ 3|windows\ 95|windows\ 98|windows\ ce|windows\ me|winht|winodws|wish|wizz|wordp|worio|works|world|worth|wwwc|wwwo|wwwster|xaldon|xbot|xenu|xirq|y\!tunnel|yacy|yahoo\-mmaudvid|yahooseeker|yahooysmcm|yamm|yand|yandex|yang|yoono|yori|yotta|yplus\ |ytunnel|zade|zagre|zeal|zebot|zerx|zeus|zhuaxia|zipcode|zixy|zmao) [NC]
RewriteRule ^.*$ - [G]

View text format

To implement the UA Blacklist, simply paste into your site’s root .htaccess file (or even better, the Apache configuration file). Upload, test, and stay current with updates and news.

So much more..

For those new to Perishable Press, please check out some of my other security resources:

• 2010 IP Blacklist
• The Perishable Press 4G Blacklist
• Blacklist Library
• HTAccess Library
• Security Library

Security is an important part of what I do around here, so please chime in with any suggestions, ideas, and comments. Thank you for visiting Perishable Press.

Source: Perishable Press

Take your WordPress skills to the next level with Digging into WordPress!

Related articles

• Over 150 of the Worst Spammers, Scrapers and Crackers from 2007
• 2010 IP Blacklist
• Disobedient Robots and Company
• How to Block IP Addresses with PHP
• How to Verify the Four Major Search Engines
• Ultimate htaccess Blacklist 2 (Compressed Version)
• Blacklist Candidate Number 2008-01-02

Read more: http://perishablepress.com/press/2010/08/09/2010-user-agent-blacklist/

Awhile ago, Silvan Mühlemann conducted a 1.5 year experiment whereby different approaches to email obfuscation were tested for effectiveness. Nine different methods were implemented, with each test account receiving anywhere from 1800 to zero spam emails. Here is an excerpt from the article:

When displaying an e-mail address on a website you obviously want to obfuscate it to avoid it getting harvested by spammers. But which obfuscation method is the best one? I drove a test to find out.

After reading through the article and its many findings, here are what seem to be the best methods for obfuscating email addresses displayed publicly on web pages..

Reverse the text direction

According to the article, one of the best methods for hiding your email address from spammers is to write it backwards and then reverse the text direction with a little CSS trickery. Let’s say we want to display and obfuscate the following email address:

This e-mail address is being protected from spambots. You need JavaScript enabled to view it

We would include the email address by writing it backwards in our web page, and wrapping it in a <span> tag with a nice class attribute:

<span class="obfuscate">moc.noitpecni@kcik</span>

Then to display it properly for our visitors, we apply the following CSS:

.obfuscate { unicode-bidi: bidi-override; direction: rtl; }

When CSS is enabled, this declaration block will reverse the display of any text within the .obfuscate-classed <span>. Works great, but not without a few downsides:

• Requires the user to manually type out the address (i.e., a mailto: link won’t work)
• The email address will display backwards if CSS is unavailable.
• When the user does a copy/paste of the email address, it’s going to be backwards.
• Bots could probably “decipher” these strings easily if programmed to do so.

Despite these issues, the reverse-text method proved 100% effective throughout the 1.5-year test, so definitely a good method to know.

Add some “null” text

Another effective way of obfuscating email information is to insert some “dummy” text into the email address itself. For example, using our “ This e-mail address is being protected from spambots. You need JavaScript enabled to view it ” address, we would insert “<span>null</span>” into the address:

<span class="obfuscate">kick@<span>null</span>inception.com</span>

..or even something more complex:

<span class="obfuscate">
	kick<span>null</span>@<span>null</span>inception<span>null</span>.com
</span>

With these injected character strings, the markup looks nothing like an actual email address. But we’re not finished with it yet – we still need to ensure that our human users are able to read the correct information. This involves hiding the nonsense <span>s with a little CSS:

.obfuscate span { display: none; }

Simple enough. Again, this works great because users will only see the email address and not the “null” text. This method is effective at hiding your email address from the spammers, but there are some familiar downsides:

• Requires the user to manually type out the address (i.e., a mailto: link won’t work)
• The email address will display the “null” text if CSS is unavailable.
• When the user does a copy/paste of the email address, it’s going to include the “null” text.
• There may be a bot that can decipher such convoluted email addresses, so be careful.

Serious accessibility/usability challenges, but still another 100% effective method according to the test results.

Encode/Decode with ROT13/JavaScript

The ROT13-algorithm is an encoding method whereby all alphabetic characters are rotated by 13. Similarly, ROT5 is used to encrypt numeric digits, whereby every number is incremented or decremented by 5. This type of cipher is commonly used in Usenet/chat threads.

Using ROT13, we can use an online tool (or PHP) to encode an email address and then use JavaScript to decode it in the web page. Encoding our example email address, we get this:

This e-mail address is being protected from spambots. You need JavaScript enabled to view it

We then include this ROT13-encoded address in the web page using this JavaScript:

<script type="text/javascript">
	document.write("<n uers=\"znvygb:
 This e-mail address is being protected from spambots. You need JavaScript enabled to view it
 \" ery=\"absbyybj\">Fraq n zrffntr</n>".replace(/[a-zA-Z]/g, 
	function(c){return String.fromCharCode((c<="Z"?90:122)>=(c=c.charCodeAt(0)+13)?c:c-26);}));
</script>

That snippet will then display the following markup:

<a href="mailto: This e-mail address is being protected from spambots. You need JavaScript enabled to view it " rel="nofollow">Send a message</a>

..which will create an email link on the page for your visitors:

This e-mail address is being protected from spambots. You need JavaScript enabled to view it

This is another 100% effective method according to the test, with the only downside being that JavaScript is required for it to work.

Other Obfuscation Methods

The test also included a fistful of other methods that varied in their overall effectiveness. Here is a quick run-down:

• Replacing “@” with “AT” and “.” with “DOT” in plain-text email addresses was also quite effective (but not 100%).
• Building/inserting the email address entirely with JavaScript was the next-best method, but some spam still got through. Looks like harvesters are learning JavaScript.
• Encoding “@” and “.” with character entities in plain-text addresses resulted in a significant volume of spam.
• Splitting the email address up with HTML comments was also ineffective at stopping spam.
• Encoding the email address with urlencode was less effective than any other method mentioned so far. Email-harvesting bots apparently have the whole “urlencode” game figured out.
• And worst of all is just using plain-text to display your email address. Definitely not recommended if you hate teh spam!

And, although it wasn’t included in the test, you could also use an image to display your email address, but the accessibility and usability is pretty poor, and there are bots that can interpret image-based text.

So which one of these email-obfuscation methods is best? One of the first three, based on the test results. Generally, these methods are useful for dropping the occasional email here and there, but perhaps the best method is to..

Use a Contact Form

When it comes to providing an easy, spam-free way for people to contact you, nothing beats the convenience of a simple, web-based email form. If you are using WordPress, there are many contact forms available, including my clean and simple Contact Coldform. Non-WordPress users also enjoy a virtual cornucopia of email-form options, including some sweet Ajax-inspired stuff.

Of course, form email has its downsides as well. Most notably, it takes longer to set up and test an online form than it does to slap down a line or two of code (as in our previous examples). Another challenging issue that I have experienced is getting contact forms to properly handle code characters (HTML, PHP, et al).

Use whatever works

If you have the time, using a web-based email form is probably the best solution for contact pages, but if you just need a way to simply include an email address, one of those first three methods may be just the ticket for keeping your publicly displayed emails spam-free.

For more information on these (and more) anti-spam email techniques, check out the original article. And, if you happen to have a favorite – and effective – email obfuscation trick, throw down in the comments and I’ll add it the post.

Source: Perishable Press

Take your WordPress skills to the next level with Digging into WordPress!

Related articles

• Industrial-Strength Spamless Email Links
• Spamless Email Address via JavaScript
• Pathetic Phishing Attempt
• Phish or Die
• Phish Phight
• Preloading Images with CSS and JavaScript
• Block Spam by Denying Access to No-Referrer Requests

Read more: http://perishablepress.com/press/2010/08/01/best-method-for-email-obfuscation/

One of my favorite security measures here at Perishable Press is the site’s virtual Blackhole trap for bad bots. The concept is simple: include a hidden link to a robots.txt-forbidden directory somewhere on your pages. Bots that ignore or disobey your robots rules will crawl the link and fall into the trap, which then performs a WHOIS Lookup and records the event in the blackhole data file. Once added to the blacklist data file, bad bots immediately are denied access to your site. I call it the “one-strike” rule: bots have one chance to follow the robots.txt protocol, check the site’s robots.txt file, and obey its directives. Failure to comply results in immediate banishment. The best part is that the Blackhole only affects bad bots: normal users never see the hidden link, and good bots obey the robots rules in the first place.

In five easy steps, you can set up your own Blackhole to trap bad bots and protect your site from evil scripts, bandwidth thieves, content scrapers, spammers, and other malicious behavior.

The Blackhole is built with PHP, and uses a bit of .htaccess to protect the blackhole directory. The blackhole script combines heavily modified versions of the Kloth.net script (for the bot trap) and the Network Query Tool (for the whois lookups). Refined over the years and completely revamped for this tutorial, the Blackhole consists of a single plug-&-play directory that contains the following four files:

• .htaccess – basic directory protection
• blackhole.dat – server-writable log file (serves as the blacklist)
• blackhole.php – checks requests against blacklist and blocks bad bots
• index.php – generates blackhole page, performs whois lookup, sends email, and logs data

These four files are all contained in a single directory named “blackhole”.

Installation Overview

I set things up to make implementation as easy as possible. Here are the five basic steps:

1. Upload the /blackhole/ directory to your site
2. Ensure writable server permissions for the blackhole.dat file
3. Add a single line to the top of your pages to include the blackhole.php file
4. Add a hidden link to the /blackhole/ directory in the footer of your pages
5. Prohibit crawling of the /blackhole/ by adding a line to your robots.txt file

It’s that easy to install on your own site, but there are many ways to customize functionality. For complete instructions, jump ahead to Implementation and Configuration. For now, I think a good way to understand how it works is to check out a demo..

One-time Live Demo

I have set up a working demo of the Blackhole for this tutorial. It works exactly like the download version, but it’s configured to block you only from the demo, not from the entire site. Here’s how it works:

1. First visit to the Blackhole demo loads the trap page, runs the whois lookup, and adds your IP address to the blacklist data file
2. Once you’re added to the blacklist, all subsequent requests for the Blackhole demo will be denied access

So you get one chance to see how it works. Once you visit, your IP will be blocked from the demo only – you will still have full access to this tutorial (and everything else). That said, here is the demo link: Blackhole Demo. Visit once to see the Blackhole trap, and then again to observe that you’ve been blocked. If I were to include the blackhole.php in the header of my theme files, you would be banned from pretty much the entire site.

Implementation and Configuration

Here are complete instructions for implementing and configuring the Perishable Press Blackhole:

Step 1: Download the Blackhole zip file, unzip and upload to your site’s root directory. This location is not required, but it enables everything to work out of the box. To use a different location, edit the include path in Step 3.

Step 2: Change file permissions for blackhole.dat to make it writable by the server. The permission settings may vary depending on server configuration. If you are unsure about this, ask your host. Note that the blackhole script needs to be able to read, write, and execute the blackhole.dat file.

Step 3: Include the bot-check script by adding the following line to the top of your pages:

<?php include($_SERVER['DOCUMENT_ROOT'] . "/blackhole/blackhole.php"); ?>

The blackhole.php script checks the request IP against the blacklist data file. If a match is found, the request is blocked with a customizable message. See the source code for more information.

Step 4: Include a hidden link to the /blackhole/ directory in the footer of your pages:

<a style="display:none;" href="http://example.com/blackhole/" rel="nofollow">Do NOT follow this link or you will be banned from the site!</a>

This is the hidden link that bad bots will follow. It’s currently hidden with CSS, so 99% of visitors won’t ever see it. To hide the link from users without CSS, replace the anchor text with a transparent 1-pixel GIF image.

Step 5: Finally, add a Disallow directive to your site’s robots.txt file:

User-agent: *
Disallow: /*/blackhole/*

This step is pretty important. Without the proper robots directives, all bots would fall into the Blackhole because they wouldn’t know any better. If a bot wants to crawl your site, it must obey the rules! The robots rule that we are using basically says, “All bots DO NOT visit the /blackhole/ directory or anything inside of it.” More on this in the next section..

Further customization: The previous five steps will get the Blackhole working, but the index.php requires a few modifications. Open the index.php file and make the following changes:

• Line #54: Edit the path to your site’s robots.txt file
• Line #56: Edit the path to your contact page (or email address)
• Lines #140/141: Edit email address with your own
• And in blackhole.php, edit line #53 with your contact info

These are the recommended changes, but the PHP is clean and generates valid HTML5, so feel free to modify the source code as needed. Note that beyond these three items, no other edits need made.

Caveat Emptor

Blocking bots is serious business. Good bots obey robots.txt rules, but there may be potentially useful bots that do not. Yahoo is the perfect example: it’s a valid search engine that sends some traffic, but sadly the Yahoo Slurp bot is too stupid to follow the rules. Since setting up the Blackhole several years ago, I’ve seen Slurp disobey robots rules hundreds of times. Bottom line: the Blackhole will block any bot that disobeys the robots.txt directives. Proceed accordingly. Update: By default, the Blackhole no longer blocks any of the popular search engines. See the next section for more information.

Whitelisting Search Bots

Initially, the Blackhole blocked any bot that disobeyed the robots.txt directives. Unfortunately, as discussed in the comments, Googlebot, Yahoo, and other major search bots do not always obey robots rules. And while blocking Yahoo! Slurp is debatable, blocking Google, MSN/Bing, et al would just be dumb. Thus, the Blackhole now “whitelists” any user agent identifying as any of the following:

• googlebot (Google)
• msnbot (MSN/Bing)
• yandex (Yandex)
• teoma (Ask)
• slurp (Yahoo)

Whitelisting these user agents ensures that anything claiming to be a major search engine is allowed open access. The downside is that user-agent strings are easily spoofed, so a bad bot could crawl along and say, “hey look, I’m teh Googlebot!” and the whitelist would grant access. It is possible to verify the true identity of each bot, but as X3M explains in the comments, doing so consumes significant resources and could overload the server. Avoiding that scenario, the Blackhole errs on the side of caution: it’s better to allow a few spoofs than to block any of the major search engines.

License and Disclaimer

The Perishable Press Blackhole is released under GNU General Public License. Check the Creative Commons for a summary and/or see the Blackhole source code for additional information. Also note that by downloading the Blackhole, you agree to accept full responsibility for its use. In no way shall the author be held accountable for anything that happens after the file has been downloaded.

Blackhole Download

Here you can download the current version of the Blackhole:

Perishable Press Blackhole for Bad Bots
    [ version 1.2 | .zip format | 5K | 10 downloads ]

Previous Versions

• Blackhole v1.0 [ version 1.0 | .zip format | 5K | 43 downloads ]
• Blackhole v1.1 [ version 1.1 | .zip format | 5K | 33 downloads ]

Source: Perishable Press

Take your WordPress skills to the next level with Digging into WordPress!

Related articles

• How to Block IP Addresses with PHP
• Blacklist Candidate Number 2008-01-02
• Blacklist Candidate Number 2008-02-10
• Blacklist Candidate Number 2008-03-09
• Blacklist Candidate Number 2008-04-27
• Blacklist Candidate Number 2008-05-31
• Blacklist Candidate Series Summary

Read more: http://perishablepress.com/press/2010/07/14/blackhole-bad-bots/