… implemented in an hour (!)

[ Discuss this post on Hacker News ]

I recently saw a nice Hacker News discussion about millionshort.com, an experimental search engine that makes it easy to search the web, minus the million most popular websites.

blekko’s search engine has a feature called slashtags, which can be used to either restrict a search to a list of websites, or remove that list of websites from the results. We typically use this feature for human curation, for example, picking out the best health websites. Hm, I thought, what an interesting hack! I’ll take that list of the most popular websites, and make slashtags which can be used to either search or exclude the most popular 10, 100, 1000, 10,000, or 100,000 websites. Our current effective limit to slashtag size is 100,000 websites, so I couldn’t do the most popular 1,000,000 sites.

The way you search with a slashtag on blekko is to add /slashtagname to your search. In this case the top-100,000 site slashtag is named /top100k, and we use a minus sign to exclude those sites:

And, after about an hour of work, voilà! Here are a couple of examples of our new bottom-website search engine in action.

great concert

This query happens to show another blekko feature in action, autoboosted slashtags. When you search for great concert, we automatically figure out that you’re interested in music, and so we use our curated /music slashtag to improve the results:

great concert

But, for the purposes of this experiment, let’s turn off this autoboost feature by adding /web on the end of our query:

great concert /web

Now let’s look at the web, minus the top 10, 100, etc. websites:

great concert -/top10 – Voilà! amazon.com disappears.

great concert -/top100

great concert -/top1000

great concert -/top10k – mtv.com and emusic.com disappear.

great concert -/top100k

hugo winner

The Hugo Award is given annually to the best science fiction book of the year. Let’s see what the bottom of the web thinks about it:

hugo winner

hugo winner -/top10

hugo winner -/top100 – flickr and about.com disappear; davidbrin.com appears.

hugo winner -/top1000 – goodreads disappears, librarything appears

hugo winner -/top10k – io9, boingboing, and Esquire disappear; dpsinfo, sfwriter.com, and mabfan.livejournal.com appear

hugo winner -/top100k – no change

Try it for yourself!

[ Discuss this post on Hacker News ]

JBofill Creative Partners made this infographic based on the recent press about blekko. This is super cool. Thanks guys!

Hitwise:

Compete.com:

“Do you know how fast you were going?” asked the Washington state trooper outside of Chehalis on my way to Seattle. “Our airplane caught you a few miles back doing 92 in a 60 zone.” Last June I rode my BMW R1200GS up to Seattle from the Bay Area to work the exhibit hall at SMX Advanced. That state trooper’s question pretty much sums up the past year at blekko for me. I was going so fast I don’t remember, but that’s not what I told the trooper.

While I was on the road promoting blekko with my trade show team, the engineers and product guys were constantly pumping out new products and features on our search engine. In January we made a bold move and banned content farms like Demand Media’s eHow from our search results, triggering our competitors to change their ranking algorithms. We followed it up in March and blocked another 1.1 million spam sites. It’s not over. The battle against spam continues daily.

Earlier in the day before I was brought down by The Man in Chehalis I was making my way from Bend, Oregon. I rode north past Mount Hood, over the Columbia River, planning to go up to the east side of Mount Ranier crossing over the pass into Seattle. Well we all know about best laid plans. About 15 miles deep into the Gifford Pinchot National Forest I came upon a large sign telling me that all the roads were closed due to snow. “Shit. I’m going to have to backtrack and take the slow road along the north shore of the Columbia River to the highway 5 slab. I’m going to be late.”

Meanwhile, back at the office, the Biz Dev team was hard at work. We inked some curation partnerships with Stack Overflow, Foodily, SB Nation and The Motley Fool to name a few. We also started powering the RSS search on Flipboard, the popular new social reader for iPad and iPhone.

The 60 mile ride along the Columbia River gorge was painfully slow and I passed the slow moving traffic whenever I had a chance; ignoring the double yellow lines. I was rewarded with spectacular views, the ones that the widest angle camera could not capture. Two hours later I jumped on to interstate 5 north, with time to make up I hit the throttle.

In late June we launched the Zorro update which included an improved index, auto-boosting slashtags and ‘3 Engine Monte.’

Riding a motorcycle on an interstate highway is about as exciting as a colonoscopy. Actually, some riders I know would disagree. So, to keep things interesting, I started to follow a white Porsche 911 driven by a guy who appeared to have purchased the car to make up for some of his inadequacies. His high levels of testosterone were obvious as he would not let me take the lead. My plan unfolded and he accelerated keeping the lead or as I call it, the bait for the radar gun. After about ten miles of cat and mouse the red led flashed on my radar detector. It appeared to be nothing.

We launched our ‘Grep the Web’ tool in September followed by good fortune, when we closed $30 million in funding with our new strategic partner Yandex leading the round with $15 million. We finished the year strong with a new index, increasing the number of slashtags we autofire to 500 and a slick new UI. All the while, staying true to our mission.

Cautiously, I slowed down and kept a tight eye on the road ahead. Five miles ahead the Porsche and I passed a trooper parked on the side of the highway. Within seconds his cherries lit up as I looked into my mirror. That sinking feeling took over my stomach as the trooper passed by me and the Porsche and signaled both of us to pull over. I was cursing to myself like a longshoreman as I navigated to a safe spot on the side of the road. He said that a airplane with radar had seen us speeding ten miles back. I did not see an airplane. He asked where I was going, and I told him, “To a trade show in Seattle.” He said the guy in Porsche was too. I was cordial and accepted the ticket for $120 which wasn’t that bad. What was bad, was that I had gotten a ticket one month earlier on highway 1 in Santa Cruz. Two tickets so close would increase my insurance rates drastically. The rest the way to Seattle and on the way home I slowed down and stuck to the speed limit. Something was wrong about that ticket and it bothered me. I did not see an airplane when my radar lit up for a second. So, the following week I hired an attorney from the Chehalis area that specialized in traffic violations to fight the ticket for me. A few months later she went to court for me and had my ticket dismissed on hearsay.

Surprisingly, speeding tickets do have a good side to them; they are telling you to slow down, smell the roses and enjoy the scenery. As the holidays are upon us and the year comes to a close, we look back at all the good times and our successes. We look forward to what 2012 will bring us. Best wishes to you from the blekko team!

Today we are releasing a major upgrade to blekko. The new blekko has significant improvements to index quality and relevance. We are also now automatically applying slashtags for over 500 categories. That means users will get the benefit of curated search results (i.e. no spam!) for over 500 search categories, regardless of whether or not they manually append a slashtag to their query.

The results in action can be pretty eye opening:

how to clean gutters
good table manners

But, as usual, don’t take our word for it – compare the difference yourself. Just do a search on blekko and add /monte to any query.

As we head into 2012, our mission at blekko is more important than ever. If you think about it, prior to the internet the newspaper was the offline equivalent of the search engine. It was the starting spot where people went to get their news, job listings, personals, movies listings, classifieds, etc. Competing editorial voices were a hallmark of competition in the newspaper business and readers benefited because they had choice.

Online, the world has changed. Search is now the access point to information. But unlike the newspaper world, there are not a lot of people competing in search. We believe this is bad for users because at the end of the day, search engines are editorial products. In the newspaper world human beings wrote the articles with their editorial voice. In the search world human beings write the algorithms that make up the search results. Currently, there is one predominant editorial voice that dictates access to information.

Our goal is to give users the benefit of an alternative editorial voice in search. What is blekko’s editorial voice? Here’s our approach:

Quality: We purposefully bias our index away from sites with low quality content.

Source based, not link based ranking: regardless of how many people link to healthspamsite.com, we believe sites like the Mayo Clinic, NIH.gov, etc. are better sites. On blekko, brands trump links.

Spam Free: blekko hates spam. We want users to get their information from quality sites designed to inform, not just monetize.

Human Curation: Their are subject matter experts in the community that will know more about specific topics than anyone in this office or any other office for that matter. We want to include these experts in our search experience, not exclude them.

Open and Transparent: blekko makes freely available to its users all of the data that provide the underpinning of our search results.

We believe that users will like the results our editorial voice delivers better than the competition. The important point though is that there is an alternative choice. The access point to information is too important and users deserve choice.

More coverage:

Venture Beat

If you didn’t finish your holiday shopping on Black Friday — opting to sleep, avoid rogue pepper spray attacks, and steer clear of looting instead — fret not for blekko makes searching for gifts easy and won’t raise your blood pressure.

Gift guides are useful but how does one sift through them when a new one pops up faster than an angel gets their bell-rung wings? Our search built-in slashtag /date helps you find the freshest content available, returning results from news articles and blogs. Enter your query, followed by /date. For instance, “2011 gift guides /date”. If you prefer to broaden your search results, enter “2011 gift guides /date /more” in the search box. Voilà, a trove of updated ideas for the hard-to-shop-for people in your life.

Being a last-minute shopper doesn’t mean you can’t be obsessive about researching products before you buy. Try out /reviews, one of many slashtags that are curated by our community of blekko users to ensure high quality results. Find reviews of this year’s coveted products like “kindle fire /reviews” or “innopad /reviews” without wasting your time wading through spam. Another handy slashtag is /gifts – whether you’re looking for fair trade or gourmet gift baskets.

Don’t have enough time to browse all these awesome websites in one sitting? You can curate your own slashtag of trusted sites and come back to it. Once you have collected your favorite urls, you can truly customize your experience by searching within your own slashtag. Check out blekko’s Help section to find out how. You can also include entire existing slashtags from blekko; for example, if you fancy /reviews as a particularly helpful tool, add “/blekko/reviews” to your slashtag. Rather than starting at square one when you come back to your task, you have a powerful and personal search engine at your service.

With holiday shopping simplified, you will have more time for ugly sweater parties, sparkling festivities, and reflecting on the year gone past.

Big day here at blekko! We are thrilled to announce that we closed our $30 million funding round and we’ve added a terrific new strategic investor to the company: Yandex.

For those of you who aren’t familiar with Yandex, it is the leading internet company in Russia, where it not only operate the country’s most popular search engine, but also the most visited website. According to TechCrunch, “in 2010 the company generated 64% of all search traffic in Russian, trumping Google.” A few months ago they completed a successful IPO. The Yandex team has been in the search business since the early 90′s and knows this industry (and what it takes to win) as much as anyone.

If their investment in this round isn’t one of the best endorsements of blekko and our mission, I don’t know what is. Our history with Yandex started at a conference in Moscow earlier this year, where our CTO Greg* was speaking. Several more meetings and two trips to Moscow later – these guys do their homework – we had a new investor.

Yandex is one of two new investors we added with this round. The other is MLC Private Equity, an Australian based investor. It’s hard to believe that after only 10 months of being live, we have enough international awareness to drive investments from Australia and Moscow. But that’s where we are – and with the completion of this round, we’re only getting started.

We have a lot more cool stuff planned for the coming months and years that we will be talking about later. But for now, we’re more than thrilled to celebrate the addition of these new partners.
___
* Greg is the one who almost accidentally pre-announced our deal at the TC Disrupt Hackathon. One of the roving reporters noticed he had these weird stickers on his keyboard and asked about them (fast forward to the 12:50 mark in the video). Greg successfully danced around the fact that they were Cyrillic stickers, as he is learning Russian.

News Coverage:
Russia’s Yandex Invests $15M in US Search Underdog – ABC News – AP

Russian Search Engine Yandex Leads $30 Million Investment In Blekko – SearchEngineLand‘

Search Engine Blekko Raises $30 Million From Russian Search Giant Yandex And Others – TechCrunch

From Russia with love: Yandex backs US search startup Blekko with $15 million, computing power – Washington Post

One of the advantages of having your own search engine is that you have access to all sorts of data that no one else does. Really, really cool data. And when you tell your friends you have all this data, you get lots of people asking you for stuff. Interesting questions like: “Can you give me a list of every site that uses Facebook Connect? In rank order?” Or: “Can you send me a list of sites that have the Google +1 button on them?”

Or, the most popular one: “Can you give me a list of every site that is running [insert name] ad network?” Obviously this is the most popular as it generates what is essentially a list of leads for someone who works for a competing ad network.

Anyway, we’re all too happy to give this info out. After all, one of blekko’s founding tenents is that the web should be open and transparent. We believe in a transparent web so much so that we wanted to extend access to this valuable data to people outside our personal social circles. So today we’re excited to announce the launching of WebGrepper. WebGrepper is a simple way to mine the web for data, that frankly, you can’t get from any other search engine.

The way it works is simple. Everyday, we will run 2 map jobs against our crawl of 4 billion pages. These will be greps for strings, patterns, regex expressions that blekko users submit to us and decide are cool. Got a grep you want to run? Submit it here. If enough people agree with you that this grep is interesting (by voting it up), we’ll run it. And we’ll post the results here. We make the top 500 results for every grep available for free to anyone who wants it. Pretty cool, eh?

We look at the web as one big, massive data set. Keyword matches (with a relevance filter) are certainly one way to pull data from that set. But its not the only way. There’s a ton of interesting data that lives within the source HTML that keywords just can’t get to. That’s where WebGrepper comes in – and now you have access to it.

So do us all a favor: think of some cool things to grep for and submit them to the community. Get out there and slash Grep the Web!

___
If you’re one of those guys who wears a t-shirt that says “Grep Me“, “Get a Grep”, “I Grep Therefore I am” or “who | grep -i blonde | date; cd“, chances are you can guess what blekko’s new Web Grepper does. For all the other folks out there who don’t speak UNIX or Klingon, Web Grepper “greps” or searches lines of code within Web files to identify relevant or matching domains based on specific topic and search terms.