Technology Assets to Support Search and Big Data Analytics Projects
Building End-to-End Solutions with Search Technologies' Assets
FILLING THE TECHNOLOGY GAP
Over the years, and through the experience of hundreds of search engine implementation projects, we have seen a number of technology challenges that available commercial or open source platforms cannot solve. To address these issues, Search Technologies has developed a collection of search engine independent technology assets to fill in the gaps in customers' search and big data analytics projects.
These assets were created by our engineers during live implementations and have evolved through numerous subsequent projects where they were used, with substantial input from our consulting customers. By eliminating the need to reinvent the wheel for some repeatable use cases like enterprise search, these assets can accelerate project timelines and reduce overall project costs, both in the short- and long-term.
Our technology assets work with a range of search engines, including SharePoint 2013, the Google Search Appliance, Amazon CloudSearch, Solr, and Elasticsearch. Each component can be deployed individually or together to complement and optimize your organization's search architecture.
- Aspire Content Processing: a content processing framework designed specifically for unstructured and semi-structured data. Aspire Content Processing has two distributions: Enterprise and Community. Aspire Enterprise is commonly used in our consulting engagements; it provides optimal functionality, a wide range of ready-made processing components, a Hadoop implementation, and distributed processing capabilities. Aspire Community is free for download by registered users.
- Content Connectors: search engine independent connectors with built-in early-binding security and metadata capture capabilities.
- Staging Repository: an intermediate repository where content can be placed after it has been extracted from a source. This staging repository allows for more efficient content reprocessing without having to reach back to the content source for every processing iteration.
- Query Processing Language (QPL): a query parsing and business rules engine, enabling sophisticated query-side processing to be set up and maintained efficiently. QPL is often deployed with the Search API Server, but also available separately in some implementations, depending on the customer's requirements.
- Search API Server: allows new endpoints to be configured in seconds. These endpoints are backed by scripts that can simply pass the incoming queries to a search engine, perform query manipulation using QPL to increase relevancy, or perform other actions such as database lookups or updates. Results are then amalgamated into a single response returned to the Search UI.
- Search UI: an end-user search interface with full source code and can be customized for specific requirements. The use of the API Server and QPL allows the Search UI to be search engine agnostic.
- Admin UI: pluggable admin interface for installation, administration, server management, and health check.
As we expand the number of our technology assets over the years, we have also developed an optimized reference architecture for creating scalable and customizable search and big data analytics applications. Built around our technology assets, open source search engines, and other complementary technologies, this proven reference architecture has enabled our customers to create working search systems more quickly and gain business value sooner.
The diagram below shows the reference architecture for a browser-based search application.
- In this example, the search application needs to access a number of disparate content sources (e.g. content management systems, text documents, e-mails, image repositories, social media sites etc.).
- Connectors acquire data from external sources. Aspire Content Processing (or some other content publishing engine) can then do the heavy lifting to prepare the content for indexing by the search engine.
- In some deployments, the Staging Repository can be a buffer between front end load and back-end publishing from the data sources.
- In this diagram, the Search API Server and QPL are deployed together to enable sophisticated query-side processing, execute scripts in "sequential parallel” fashion, and serve relevant results to the Search UI. In some cases, QPL is available separately based on the customer's implementation needs.
- Search UI provides basic or custom templates for most search use cases, including e-commerce, corporate wide search, data warehouse analytics, media & publishing, recruiting, and many others.
- Admin UI is a central, customizable dashboard that allows system admins to holistically configure, manage, and monitor all system components.
- When application requirements call for text mining, machine learning, semantic analysis, or quality metrics, it may be necessary to deploy Aspire in a big data array as part of a Big Data Framework (Hadoop). What may not seem like a big data job (millions of documents) can quickly become one when advanced text mining is required (billions of words, phrases, and semantic relationships).
- Create an end-to-end search system in days, not weeks, and bring immediate business value
- Provide a risk-free, cost-effective approach to switching between search platforms, such as replacing the Google Search Appliance, migrating from FAST ESP to SharePoint, moving from Solr to Elasticsearch, etc.
- Fill in the gaps in open source and commercial search engine platforms using search engine independent, complementary technology assets
- Deployable on-premise or in the cloud
When appropriate to the solution architecture, our technology assets facilitate the efficient delivery and support of custom search solutions, and our customers benefit from increased reliability, as well as cost savings. We propose using these assets only where there are clear and specific benefits to the project, and any decision to use them is made by our technical team in full consultation with the customer.
Some of our technologies can also be licensed by non-consulting customers and OEMs. In these cases, we offer support and maintenance services for solutions that deploy our technology assets. Support for other solution components (commercial and open source) is also available from our partners, or from Search Technologies.
FOR MORE INFORMATION
Contact us for more information about our technology assets and how they can help optimize your search and big data analytics applications.