![\"pipeline](\"https:\/\/2thenew.today\/blog\/wp-ttn-blog\/uploads\/2025\/08\/Screenshot-from-2025-08-23-20-07-35.png\")
pipeline flow diagram<\/p><\/div>\n
Setting Up the Basics<\/h2>\n
We\u2019ll use:<\/p>\n
- \n
- MySQL as the data source<\/li>\n
- Elasticsearch (v8+) as the search engine<\/li>\n
- Python + official Elasticsearch client for migration<\/li>\n<\/ul>\n
![\"secure](\"https:\/\/2thenew.today\/blog\/wp-ttn-blog\/uploads\/2025\/08\/Screenshot-from-2025-08-23-18-09-56-1.png\")
Here we set up secure connections to both MySQL and Elasticsearch<\/p><\/div>\n
Deleting Old Indices<\/h2>\n
Before creating new indices, it\u2019s a good practice to remove any outdated ones. This prevents mapping conflicts or duplicate records.This ensures your migration always starts on a clean slate.<\/p>\n
![\"Delete](\"https:\/\/2thenew.today\/blog\/wp-ttn-blog\/uploads\/2025\/08\/Screenshot-from-2025-08-23-19-59-43.png\")
Delete index Data<\/p><\/div>\n
Creating Elasticsearch Mappings<\/h2>\n
Unlike MySQL, Elasticsearch requires you to define how fields should be indexed. For our company_registry, we use the following mapping:<\/p>\n
![\"company](\"https:\/\/2thenew.today\/blog\/wp-ttn-blog\/uploads\/2025\/08\/Screenshot-from-2025-08-23-19-40-34.png\")
company registry index<\/p><\/div>\n
This ensures that:<\/p>\n
- \n
- companyName supports both full-text and exact match queries.<\/li>\n
- excludeFromSearch is a boolean flag.<\/li>\n
- id is stored as a numeric long.<\/li>\n<\/ul>\n
Transforming the Data<\/h2>\n
Raw MySQL data doesn\u2019t always map directly to Elasticsearch. We normalize fields before indexing:<\/p>\n
![\"Transforming](\"https:\/\/2thenew.today\/blog\/wp-ttn-blog\/uploads\/2025\/08\/Screenshot-from-2025-08-23-19-43-38.png\")
Transforming Data<\/p><\/div>\n
Fetching and Indexing Data<\/h2>\n
Here\u2019s the migration logic for company_registry:<\/p>\n
- \n
- Fetch rows from MySQL<\/li>\n
- Apply transformations<\/li>\n
- Push them into Elasticsearch in bulk (faster than single inserts)<\/li>\n<\/ul>\n
![\"Fetching](\"https:\/\/2thenew.today\/blog\/wp-ttn-blog\/uploads\/2025\/08\/Screenshot-from-2025-08-23-19-46-58.png\")
Fetching and Indexing Data<\/p><\/div>\n
Running the Migration<\/h2>\n
Finally, execute the migration:<\/p>\n
![\"Running](\"https:\/\/2thenew.today\/blog\/wp-ttn-blog\/uploads\/2025\/08\/Screenshot-from-2025-08-23-20-06-19.png\")
Running Migration<\/p><\/div>\n
When run, this script will:<\/p>\n
- \n
- Delete any old index (if needed)<\/li>\n
- Recreate the index with proper mappings<\/li>\n
- Migrate rows from MySQL into Elasticsearch<\/li>\n<\/ul>\n
Key Lessons Learned<\/h2>\n
- \n
- Always define explicit mappings otherwise ES may misinterpret your fields.<\/li>\n
- Transform booleans & dates \u2014 MySQL often stores them as TINYINT or DATETIME, which ES won\u2019t parse correctly without conversion.<\/li>\n
- Use bulk indexing for performance.<\/li>\n
- Index per table \u2014 each MySQL table can map to its own index (e.g., company_registry, user, etc.).<\/li>\n<\/ul>\n
Conclusion<\/h2>\n
By building a simple migration pipeline with Python, we successfully moved company_registry data from MySQL into Elasticsearch. Once indexed, the data can be leveraged for autocomplete, fuzzy search, typo handling, and blazing-fast queries \u2014 things MySQL alone struggles with.This approach is flexible: just extend the transformation functions for other tables like user or company_order and you have a reusable ETL pipeline for all your entities.<\/p>\n","protected":false},"excerpt":{"rendered":"
Introduction Elasticsearch has become a go-to choice for building fast, intelligent search experiences. But what if your source of truth is a relational database like MySQL? In this blog, we\u2019ll walk through how we migrated structured relational data into Elasticsearch using Python \u2014 with company_registry as a working example. Why Migrate from MySQL to Elasticsearch? […]<\/p>\n","protected":false},"author":2132,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"iawp_total_views":60,"footnotes":""},"categories":[446],"tags":[5388,7788,1524,4844,76,1358,332],"class_list":["post-74122","post","type-post","status-publish","format-standard","hentry","category-java","tag-dataengineering","tag-datamigration","tag-elasticsearch-2","tag-java","tag-mysql","tag-python","tag-search"],"aioseo_notices":[],"aioseo_head":"\n\t\t\n\t\n\t\n\t\n\t\n\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t