Inception (noun) an event that is a beginning; a first part or stage of subsequent events
Two problems addressed by Inception
Data harvesting will keep increasing for years to come.
Harvested data is mostly stored in non-relational databases in cloud space.
Most harvested data is not used for lack of data mining software, and limited execution speed of mining software.
Private harvested data is increasing stolen, used and sold without permission.
Backing up files can result in a digital hoarding problem.
Information within hoarded documents is often forgotten.
Shared directories containing hoarded documents causes a security and privacy nightmare.
Theft occurs when access is provided to digital hoards within organizations.
Theft is for personal, corporate and government gain, where some theft is to increase personal digital hoards.
About Big Data Hoarding
- Most web sites, applications and operating systems hoard private data without asking, and transport data to remote networks for storage and use.
- Governments hoard through backbones, and backdoors into operating systems, applications, and ISPs.
- Many countries pass backbone data to other aligned countries to hoard before it is passed back into the country of origin.
- This vast movement of data bypasses surveillance laws of people and organizations within each country.
- The rate of big data hoarding has been accelerating for years.
- This form of hoarding cannot be stopped because of the wealth and control associated with big data and AI.
- Most big data has not, and will not, be processed due to bottlenecks associated with data mining.
- This makes the data close to useless as it ages without use.
- This risky spread of unlimited use, and insecure storage, of private data is completely unacceptable.
Our solution
- The Inception service and API rapidly transforms raw data and documents into other desired forms.
- Inception is an automated data pre-processor that can greatly speed up data transformation.
- A student could create processing scripts used to transform data into desired forms.
- There is no need to craft extensive data-mining software that is written in interpreted languages.
- Local private data can remain local, private and secure after transformation.
- The Inception service has been customized to automate transformation of Microsoft Office documents.
About automated processing of Microsoft Office documents
Inception and database services execute in the background
- The Inception service.
- Automatically detects and transforms Word, Excel and PowerPoint documents into full text, then words by sequence and group.
- Documents and associated words are recorded in a memory-based database.
- When directory processing completes then the Inception memory-based database is committed to local storage.
- The Database microservice.
- Starts by passing an encrypted file containing database-related information plus preferences.
- Automatically loads the Inception database into memory.
- Each memory-based database is written to a PostgreSQL relational database along with full document text.
- Options can be set to enforce database security and privacy within encrypted file passed to start the service.
- Encrypt full text before it is committed to the database.
- Legitmate use permitted while use of text on database breach is halted.
- Words plus user-defined salt are hashed before being committed to the database.
- Legitmate use permitted while use of words on database breach is halted.
- Private Intranet software is used to perform searches of words and phrases to locate documents for review.
Evolution
The process of transforming digital data into new forms is as old as the computer industry.
Inception was created, recreated, rewritten and perpetually enhanced over many years.
- Earliest use was to transform magazine editorial submissions into formats used to review, edit and typeset.
- We created a commercial application that allowed selection and viewing of word processing, spread sheet and presentation documents.
- The application decoded file headers to determine file type, then transformed each file to present to users.
-
Large amounts of mainframe data stored on Magnetic Optical media was directly read using low-level code under Windows.
- The English and French data was automatically retained as full text plus transformed into words and phrases.
- We created an advanced indexing system used to quickly locate the words and phrases from slow media.
- This allowed users to rapidly search, view, export and print English and French text used during document translation.
- Legacy DB2 and Informix RDMS database exports were transformed to import into an Oracle RDMS for use through the Siebel CRM.
- Thousands of Word and PDF documents were automatically transformed into HTML and XML formats for technical and developer support sites.
- Automated transformation of bloated, unconventional XML to clean, DTD-validated XML for linguistic translation.
- Automated transformation of email real estate leads using many processing scripts to create SQL scripts for automated import into an MySQL database.
- Signature analysis was used for several lead sources with many sub-types to select a processing script .
- Automated transformation of Microsoft Office Word docx, Excel xlsx and PowerPoint pptx documents.
- Multi-threaded automation provides processing of complex directory chains along with individual documents.
- Resulting UTF-8 text is broken into words with stop-words removed.
- Pertinent information that includes document size, timestamps, word sequence and groups, is retained in a memory-based database.
- The memory-based database is written to local storage after processing completes.
- A database service handles writing the results to a relational database.
Refactored
Inception was refactored and largely rewritten in a selective mixture of C and C++.
We have a great deal of experience developing software in several low to high-level languages.
We could have quickly rewritten Inception in higher-level languages.
We chose to take the difficult route of coding in C and C++.
Refactoring Inception in any of the higher-level languages would have saved us a great deal of time and cost.
We could have made use of open-source applications to save us even more time and cost.
Use of open-source applications that are usually written in higher-level languages, could:
Inception was refined to achieve the highest processing speed.
We chose to develop key-portions of Inception in C to directly load and process files in memory because it is fastest and provides greatest control.
We did not use C++ memory handling because it makes indirect use of memory, which is slower than direct access.
Direct memory access is faster in situations where CPU caching is ineffective due to multi-threaded processing.
- Each processing thread repeatedly changes their own memory buffers in different sections of the buffers.
- Thread memory is discarded once processing ends.
We did not use C++ vectors and other associated functions.
- Each function controls memory allocation, expansion and deallocation.
- This is an unpredicatable and uncontrollable behaviour that will reduce processing speed.
The speed difference between direct and indirect memory access becomes noticable when processing thousands to millions of documents.
We created Inception for speed, stability and user-control.
We utilized the C++ object-oriented framework to make it easier to isolate functionality and maintain Inception.
The Inception code base is well constructed, well commented and documented, and fairly easy to understand.
We can enhance Inception to automate processing of other document types as needed in future.
- The scripting command set has been expanded.
- An Application Programming Interface (API) is provided through use of Linux shared and static libraries.
- Use of the API enables custom software to transform data into new forms by making use of Inception.
- API functionality is being added to encypt, and subsequently decrypt, data returned by the API.
- Adjustable async process threading has been added.
- Automated directory processing has been added.
- A memory-based NoSQL engine was created and used to retain processed documents and associate words.
- Use of the memory-based database accelerates document processing speed.
- The memory-based database is written to local storage after directory processing has been completed.
- A database service reloads the memory-based database and securely writes it to a relational database.
Automation
Inception automates document transformation.
It does so from within custom software that utilizes the Inception API.
It does so when used as a Linux service that automates document processing.
- Files and directory chains are transferred to an Inception input queue.
- Each are automatically processed in FIFO sequence.
- Automatic detection of file type, then document type, results in correct processing.
- The processing speed reduces the total threads that are required for processing.
Shared features of Kryptera
Files and directory chains transferred into an input queue must be fully written before use.
This feature is critical for document processing automation.
Without validation, file and directories could be moved or loaded into memory while still being written.
This problem was discovered while developing the Kryptera HSM.
Tracing down a solution required low-level R&D that was specific to Linux.
A variable delay is placed after a file or directory chain is completely written.
The delay is only needed if cached writes occur after file writes are complete.
Operating Platform
Inception builds and runs on Debian and related trees such as Ubuntu and Devuan, plus RedHat, Centos and Fedora.
It would be fairly simple to refine Inception to build and execute under Unix.
Inception will run as an initd or systemd service, a standalone service-like application, or a standalone application to create processing scripts.
Inception shared and static libraries can be used by custom software to automate document and data transformation.
We chose to develop Inception to operate under Linux rather than Windows to ensure the highest processing speed and stability.
Windows, applications and services utilize memory, CPU cores, file storage and network bandwidth which would reduce Inception processing speed.
Refining Inception to build and execute under Windows will occur in the near future.
Configuration
Inception service startup flags.
./ServiceInception -folder_input input/ -folder_output output/ -folder_process process/ -folder_script script/ -folder_error error/ -folder_log log/
This starts Inception as a standalone application which can also be used as a systemd service.
-start_daemon is passed to start Inception as a background initd daemon (service). The design allows shared directories to be used for input and output queues, and allows use of higher-speed storage for thread-processing space.
PostgreSQL database microservice startup flags.
./ServicePostgreSQL -folder_input input/ -folder_log log/ -encrypted_auth PATH_FILENAME
Pass -start_daemon to start the service as a background initd daemon. Set -folder_input to what -folder_output is set to for Inception.
A utility is provided to encrypt database-related information and preferences.
Script Creation
Inception Script Creation Flags.
./ServiceInception -test_source path/source_file
-test_target path/target_file
-test_script path/script_file
The flags are used to test creation or refinement of a processing script.
Load the source, script and target files into a text editor.
Change the script then start command line processing to update the target for review.
Scripting
A key Inception feature is user-defined processing scripts that are used to automate document transformation into desired formats.
The Inception scripting language contains many commands that can be separated into groups:
hide <tag_open>data</tag_open>,
move down till Tag Level == tag_level_to
then start a new block using the passed parameters
making sure to include the extracted data.
Our scripting language can be extended in future to better process any form of data plus other document formats.
Microsoft Office Scripting
We added functionality to automate transformation of Microsoft Office documents into words that are inserted into a relational database.
- A five line script that only uses three commands is used to transform Word XML documents into text.
- A six line script that only uses four commands is used to transform Excel XML documents into text.
- A six line script that only uses four commands is used to transform PowerPoint XML slides into text.
Application Programming Interface (API)
Five API functions are provided that enable custom software to automate document transformation.
The functions differ on the data that is passed to and from the API.
API naming is based on the International Civil Aviation Organization (ICAO) alphabet.
Alpha
/**
* \brief Load source and processing script, write target, deallocate source and script memory, caller has target file.
*
* Buffers pSourcePath & pProcessingScript, calls Delta, writes pTargetPath,
* deallocates pSourceBuffer & pProcessingScriptBuffer, does not return pSourceBuffer.
*
* \param pSourcePath
* \param pTargetPath
* \param pProcessingScript
* \param pSizeSourceBuffer pointer to the size of the pSourceBuffer
* \return FAILURE_OKAY on success else various FAILURE_* on error
*/
int32_t Alpha( char *pSourcePath, char *pTargetPath, char *pProcessingScript, uint32_t *pSizeSourceBuffer );
Bravo
/**
* \brief Load processing script, write target, deallocate script memory, caller has target file and deallocates pSourceBuffer.
*
* Buffers pProcessingScript, calls Delta, writes pTargetPath,
* deallocates pProcessingScriptBuffer, does not return pSourceBuffer.
*
* \param pSourceBuffer
* \param pTargetPath
* \param pProcessingScript
* \param pSizeSourceBuffer pointer to the size of the pSourceBuffer
* \return FAILURE_OKAY on success else various FAILURE_* on error
*/
int32_t Bravo( uint8_t *pSourceBuffer, char *pTargetPath, char *pProcessingScript, uint32_t *pSizeSourceBuffer );
Charlie
/**
* \brief Load processing script, deallocate script, returns target (pSourceBuffer), caller writes target
* and deallocates pSourceBuffer.
*
* Buffers pProcessingScript, calls Delta, deallocates pProcessingScriptBuffer, returns altered pSourceBuffer.
*
* \param pSourceBuffer
* \param pProcessingScript
* \param pSizeSourceBuffer pointer to the size of the pSourceBuffer
* \return FAILURE_OKAY on success else various FAILURE_* on error
*/
int32_t Charlie( uint8_t *pSourceBuffer, char *pProcessingScript, uint32_t *pSizeSourceBuffer );
Delta
/**
* \brief Returns target (pSourceBuffer), caller has target (pSourceBuffer) and deallocates pSourceBuffer
* and pProcessingScriptBuffer.
*
* Delta is the class called by all others.
*
* \param pSourceBuffer
* \param pProcessingScriptBuffer
* \param pSizeSourceBuffer pointer to the size of the pSourceBuffer
* \param pMaxSourceBuffer: size of pSourceBuffer when malloc'd
* \return FAILURE_OKAY on success else various FAILURE_* on error
*/
int32_t Delta( uint8_t *pSourceBuffer, uint8_t *pProcessingScriptBuffer, uint32_t *pSizeSourceBuffer, uint64_t pMaxSourceBuffer );
Echo
/**
* \brief processing script is populated, load source, write target, deallocate source memory, caller has target file.
*
* Buffers pSourcePath & pProcessingScript, calls Delta, writes pTargetPath,
* deallocates pSourceBuffer & pProcessingScriptBuffer, does not return pSourceBuffer.
*
* \param pSourcePath
* \param pTargetPath
* \param pProcessingScriptBuffer
* \param pSizeSourceBuffer pointer to the size of the pSourceBuffer
* \return FAILURE_OKAY on success else various FAILURE_* on error
**/
int32_t Echo( char *pSourcePath, char *pTargetPath, uint8_t *pProcessingScriptBuffer, uint32_t *pSizeSourceBuffer );
Virtualized
Inception can be deployed on a virtualized server such as Docker.
Docker containers are easy to create, configure and use.
Docker supports Volumes which enable use of Inception input and output queues.
Kubernetes can be used to control 1-n Docker containers.
Processing Speed
Intel Core i3-4150 4 CPU cores @ 3.50 GHz / 800 MHz,
1 TB 7200 RPM SATA HDD, 8 async threads limit.
Documents transformed into text.
- 408 Word .docx files, 113.8 MB, 9.754233 seconds.
- 484 Excel .xlsx files, 108.2 MB, 99.723453 seconds.
- 132 PowerPoint .pptx files, 145.34 MB, 8.116166 seconds.
Documents transformed into text, split into words, retained in memory, with memory tables saved as binary files.
- 408 Word .docx files, 113.8 MB, 11.513733 seconds, 147,769 words.
- 484 Excel .xlsx files, 108.2 MB, 102.881834 seconds, 1,235,563 words.
- 132 PowerPoint .pptx files, 145.34 MB, 8.150428 seconds, 6,267 words.
PostgreSQL database microservice loading saved tables into memory, then writing results to a relational database.
- 408 Word .docx files, 113.8 MB, 65.207372 seconds, 147,769 words.
- 484 Excel .xlsx files, 108.2 MB, 524.543025 seconds, 1,235,563 words.
- 132 PowerPoint .pptx files, 145.34 MB, 15.441017 seconds, 6,267 words.
Current and Future Enhancement.
- Inception achieves the highest speed of transforming documents into words.
- Writing documents information and associated words to a relational database greatly increases directory processing time.
- Database activity has been separated into a unique microservice.
- Inception achieves the highest processing speed where writing results to the database occurs at a slower pace.
- The database service will be enhanced in future to support additional database systems without impact on the Inception service.
- Inception will be enhanced in future to process additional document types that include PDF and LibreOffice documents.
Some of our Competition
| Alteryx |
- Unlock Data from PDFs and Images.
- Extract data encoded in system-generated PDFs with PDF to Text and leverage Google Tesseract’s powerful OCR (Optical Character
- Recognition) capabilities to extract image content from documents (e.g. invoices, business cards, forms, etc.) with Image to Text
- Develop image classification models and apply those models to new images with Image Recognition
- Automatically extract entity names from text using Named Entity Recognition (NER)
- Because Data-Driven Decisions Matter Now More Than Ever
- Empower everyone to generate the insights you need, when you need them.
- Trusted by the World’s Most Influential Brands
- Our customers understand the power of data and analytics.
- That’s why they chose Alteryx to help empower their employees to make transformational outcomes.
| ShinyDocs |
- Their product processes documents within shared directories into text, then words, stored in a database.
- Support is provided for Microsoft Office Word, Excel annd PowerPoint documents, and PDF documents.
| OpenRefine |
- Clean and transform data from one format to another.
- Free, open source, written in Java, works using a browser.
- Poorly constructed user interface, complex to use.
- Book written to use “Using OpenRefine” plus complex documentation online.
| Trifacta Wrangler |
- Venture started by makers of Data Wrangler
- Interactive tool for data cleaning and transformation.
- Less formatting time and larger focus on analysing data.
- Machine learning algorithms suggest common transformations and aggregations.
- Free.
| Drake |
- Simple to use, extensible, text based data workflow management.
- Has data processing steps defined along with their inputs and output.
- Can automatically resolve dependencies, calculate the command to execute in order.
- Organizes command execution around data and its dependencies.
| TIBCO Clarity |
- Data cleaning software-as-a-service.
- Can validate data, perform deduplication, cleanse addresses.
- Helps quickly identify trends and make smarter decisions.
- Standardises raw data from disparate sources.
- Provides good quality data for accurate analysis.
| Winpure |
- Clean large amount of data, fuzzy matching, remove duplicates, correct and standardize.
- Support of databases, spreadsheets, CRMs and text files.
| Data Ladder |
- DataMatch and DataMatch Enterprise
- Advanced fuzzy matching algorithms for up to 100 million records.
| Quadient Data Cleaner |
- Data profiling engine for analyzing data quality.
- Finds missing values, patterns, character sets and other characteristics in a data set.
- Detects duplicates using fuzzy logic.
- Build custom cleansing rules and compose into scenarios to target databases.
| Cloudingo |
- Salesforce data cleansing tool
- Eliminates duplicates, cleans records, and maintains data quality.
- Data is updated in bulk, and imported files are cleansed before accessing Salesforce.
- Automation capabilities regularly scan data for errors.
- Features deleting unnecessary and stale records, update records in bulk, automate on a schedule.
| Reifier |
- Utilises Spark for distributed entity resolution, deduplication and record linkage.
- It uses machine learning algorithms to provide entity resolution and fuzzy data matching.
| IBM Infosphere Quality Stage |
- Database cleansing and management.
- Builds consistent views of customers, vendors, products, locations, etc.
- For big data, business intelligence, data warehousing, master data management, etc.
Inception is designed to fully automate conversion of documents into desired formats.
- This includes automated processing of complex directory chains.
- Inception processes documents at the highest possible speed.
- Inception will perform in a straightforward manner 24/7.
- None of the competitive products provide similar features.
- All products are complex in construct and use.
- No competitive product makes use of a processing command set that allows users to custom scripts to transform their data.
- Some of the competitive products utilize open source products to process documents into text.
- Use of open-source applications that are usually written in higher-level languages, could:
- Reduce processing speed.
- Eliminate key-features such as directory processing.
- Incur stability, security and privacy risks.
- Cause maintenance and liability problems.
Summary
Inception solves data hoarding issues through rapid automation of document transformation.
Transformed Office documents are parsed, with words separated then separately imported into a database for indexing.
Automated processing of shared directory chains ensures a faster return on investment for clients.
Copyright © Kryptera