Following our earlier blog on the history of search engines from 1990 to 2000, it’s time to take a look at how they have advanced in the 21st century.
Search engines have come a long way since Archie made its debut in 1990. Today, they are even more vital to help people locate information on an estimated 30 trillion pages on the web.
A good search engine is defined by its usability, according to Jakob Nielsen, a Danish software engineer and web consultant. Users should be able to accomplish their task quickly and easily, even if they have never used the search engine before. It should also provide user satisfaction and relevant results.
© Rawpixel.com / Adobe Stock
Subject Specific Popularity
Further developments began in 2000, when the Teoma search engine made its debut. It was the first to use clustering to organise sites by Subject Specific Popularity – this means they try to find local web communities. Ask Jeeves bought Teoma in 2001, replacing the existing DirectHit search technology.
Yahoo bought Inktomi in 2002, followed by Overture Services Inc in 2003. The latter had already bought AlltheWeb and Altavista, prior to being taken over by Yahoo. In 2003, Yahoo began using its Yahoo Slurp web crawler. Yahoo Search then combined the technology from all of its acquisitions. Until this point, Yahoo had been using Google to power its searches.
In November 2004, Microsoft started to use its own indexer and crawler for MSN Search. Prior to this, it had used blended results from Inktomi and LookSmart. In January 2005, Google, Microsoft and Yahoo collectively introduced new webmaster tools, the nofollow attribute, to combat link spam.
Bill Gross, the owner of Overture Services Inc, launched the Snap search engine in October 2005. It had many innovative features, including a display of search volumes, auto-completion and related terms. However, it went out of business relatively quickly.
In 2006, Wikia launched Wikia Search, a new search engine run by human curation. It was proposed on 23rd December 2006 and was tested privately on 24th December 2007. In August 2008, the toolbar made its public debut, but it had shut down by May 2009.
On 28th January 2008, Cuil (a web search engine created by ex-Google developers) was launched. It used picture thumbnails to display search results. However, it closed down on 17th September 2010.
Known as DDG, DuckDuckGo made its debut on 25th September 2008. The web search engine protected users’ privacy by not profiling the searchers. It shows all users the same search results for a specific search term, showing the best results from more than 400 individual sources, including other search engines such as Yahoo and Bing and crowdsourced sites such as Wikipedia. It had 36.2 million daily direct searches in March 2019.
On 29th July 2009, a major search engine consolidation took place, with Microsoft and Yahoo announcing they had made a 10-year deal to replace the Yahoo search engine with Bing. Yahoo was to keep 88% of the revenue from the search ad sales for five years and was permitted to sell adverts on some Microsoft sites. The Yahoo Search would retain its branding, but would ultimately feature the “Powered by Bing” strapline.
Google launched Google Instant in September 2010, a predictive feature. As users typed, Google predicted their whole search query, with the same technology used for Google Suggest. It later became the autocomplete feature. It showed the results instantaneously for the top prediction. Google claimed it would save two to five seconds on each search query.
A new search engine, Blekko, was launched on 1st November 2010. It used slashtags to enable users to search in more targeted categories. It promised a web search engine that would provide “better search results than Google Search”. In March 2015, it was acquired by IBM, but the service was later discontinued.
On-going updates to Google algorithms have affected user experience and SEO. In February 2011, Google Panda was launched. Its goal was to continue stamping out spam, content farms and sites with a high ad-to-content ratio.
In January 2012, Google launched Search Plus Your World. This was a deeper integration of users’ social data into the search, followed by its “Webspam update” (later known as Penguin), which decreased the rankings of search engines that failed to follow Google’s webmaster guidelines.
Google began to roll out Knowledge Graph in May 2012, which stored semantic relationships between objects internally. Google then began to display supplementary information on the side about objects relating to the search queries.
Hummingbird was released in August 2013. The algorithm update was aimed at enabling more effective use of the Knowledge Graph. In May 2014, Google implemented its PayDay Loan update, particularly to target spam websites.
Further updates continued throughout 2014, including Pigeon in July, which improved Google’s distance and location ranking parameters, followed by updates to Panda in September and Penguin in October.
A large number of algorithm updates have taken place regularly over the past four years. Some major updates have been confirmed by Google, while other suspected updates have never been confirmed. Users believe the updates are due to search ranking volatility experienced in clusters on specific dates.
Recent major updates have included Fred on 7th March 2017, which had a massive impact on search results. It appeared to target low-value content, although Google has never revealed the specifics.
In March 2018, Google confirmed its broad core algorithm update. It was aimed at benefiting under-rewarded pages and websites were urged to “continue building great content”. Further broad core algorithm updates were released last year, aimed at improving content.
The latest confirmed update took place on 12th March 2019, when a core update, known as Florida 2, was described as “one of the biggest Google updates in years” by Google’s Search liaison, Danny Sullivan. Broad core updates improve Google’s overall algorithm, so that search queries are better understood. They help Google to match search queries to web pages more accurately to improve users’ satisfaction.
Future of search engines
Although search engines have been under continual development for the past two decades, there’s still room for improvement. Industry insiders believe artificial intelligence will play an increasing role in search engines, enabling them to better understand the query to improve the users’ experience.
In Japan, experiments are taking place in which bots are attempting to write web content. The first effort made it through the first round of a national literary competition! Although it’s unlikely to be as good as anything an experienced journalist can write, it has been suggested that bot-written content may be a viable option for businesses’ SEO campaigns.
However, critics are sceptical and argue that although AI may help in terms of speed (giving humans more time to add the creative touches), however much we try to give bots emotional intelligence, at the end of the day, they’re just not human!
The World Wide Web today is crucial to our daily life, in both business and social terms, with search engines playing a key role in the user experience. Please contact The Cornwall SEO Co. to find out about how our SEO expertise could help your business.