Into The Source: Data lake planning, migration, and delivery through cloud native consulting
Meet Into The Source
Into The Source is a digital agency that helps companies, brands, and organizations grow by delivering full-service software solutions. It leverages its creativity, data, knowledge, and expertise to create bespoke and user-centric approaches to meet the complex requirements of each of its clients.
Into The Source prides itself on building data-driven digital solutions, believing that delivering powerful digital experiences leads to customer loyalty. Therefore, it ensures that every online interaction it develops is compelling and drives conversions. Its client base ranges from sports clubs and hardware stores to construction and retail organizations. Into the Source provides various custom solutions for business analysis, UX/UI design, SEO, software development, and integration.
Into The Source is a full-service agency that values proactive and results-oriented methods that deliver long-term and lasting business objectives.
The Challenge
Into The Source was tasked with building and implementing a highly complex data lake project for its long-term client, Vadain, the international market leader in custom-made curtains, with a focus on real-time visibility and predictable scale.
However, Into The Source faced the challenge of supporting its customer, Vadain, in managing larger, more complex programs with confidence and governance. Vadain required a highly versatile and robust IT platform capable of handling extensive customization and near real-time order tracking across their entire process. Their operations involved a labour-intensive manufacturing process with multiple teams, disciplines, and data formats, making digitization and streamlining difficult.
To meet these demands, Into The Source needed to develop a bespoke cloud-based platform from scratch, capable of managing vast amounts of data, processing thousands+ of orders per week, and providing accurate tracking and forecasting to drive faster, more confident decisions.
The Solution
Over the years, Into The Source has been actively leveraging the team of expert Cloud Reliability Engineers (CREs) at DoiT for technical support and optimization across multiple projects, the largest and most notable being the Vadain data lake project.
In addition, DoiT also supported Into The Source with its on-premises systems’ AWS interoperability and laid the foundations for future compliance frameworks.
“We have bi-weekly meetings with DoiT and AWS in which we discuss our requirements and wishes – DoiT then provides us with proactive solutions, allowing us to offer rapid turnaround times for our clients.” – Tim Vogt, Into The Source Lead Developer.
Data lake planning and migration with the expertise from DoiT
Into The Source relies on DoiT and its expert cloud knowledge for specialized advice and support during production. As a technical partner, DoiT has helped educate and upskill the team, empowering them to implement solutions independently. This proactive collaboration was especially valuable during the Vadain data lake project.
To build a scalable and robust AWS platform that met the project requirements, DoiT began with an initial assessment and planning phase. This phase involved understanding the platform’s needs and aligning the data lake’s goals with all stakeholders. In collaboration with DoiT, Into The Source started outlining the AWS infrastructure while leveraging DoiT to develop an architectural diagram for the data lake with clear runbooks and guardrails. This crucial phase helped identify key components such as data ingestion, storage, processing, and analytics while highlighting risk mitigation strategies. Before proceeding, the team clearly understood scalability, flexibility, compliance, and security needs.
Next, the DoiT team focussed on the migration phases. Utilizing AWS Database Migration Service (DMS), the DoiT team assisted Into The Source with migrating a large amount of data from the data warehouse to ensure on-premise compatibility. The team meticulously planned each phase over several months, with clearly defined milestones throughout the successful migration of the databases. The DoiT team supported Into The Source, ensuring they were working according to AWS best practices while considering future compliance frameworks and audits.
DoiT helped Into The Source navigate multiple challenges and hurdles, including processing and importing data ingestion pipelines from various sources to make them accessible and usable. Additionally, the DoiT CREs supported Into The Source with data security and compliance, handling multiple schema changes throughout the migration process and maintaining the data lake’s performance and scalability.
DoiT helped Into The Source navigate multiple challenges and hurdles, including processing and importing data ingestion pipelines from various sources to standardize access and usability.
Immersion sessions and upskilling with experts at DoiT
The Cloud Engineers at DoiT conducted a series of “Workshops” and took time to help bridge specific knowledge gaps within the team. This has been crucial in equipping Into The Source’s development team with the necessary tools to provide ongoing fixes and proactively make AWS optimizations.
“Working with DoiT has empowered our internal teams and given us the long-term tools to achieve our client’s goals and objectives for similar projects in the future” – Tim Vogt, Lead Developer at Into The Source.
Using DoiT Technology for cloud optimizations, usage analytics, and insights
While the data lake project is notable for its scale, DoiT has continuously supported Into The Source, specifically with live dashboards for usage analytics, insights, and ongoing cloud optimizations.
As an automated solution, DoiT Cloud Intelligence™ has given Into The Source access to easy-to-implement recommendations, enabling the team to optimize pricing, eliminate waste and redundancy, and better align requested resources to actual usage.
“With DoiT technology, we’re able to get much better insights into costs and optimizations, not only for us as individual developers, but as a business – understanding areas we’re potentially overspending or are inefficient” – Tim Vogt, Lead Developer at Into The Source.
The Results
The support from DoiT has empowered Into The Source’s developers, enhancing their AWS knowledge and capability to manage and maintain complex data lake processes with confidence. The rapid and expert assistance provided by DoiT has been crucial in fixing breaks and upskilling the team. On average, Into The Source has reclaimed ~75 minutes per support ticket (51 hours YTD), freeing engineering capacity for higher-value work. This efficiency gain has allowed Into The Source’s team to focus more on strategic tasks, enhancing their productivity and overall operational efficiency.
“We’ve really benefited from the collaboration with DoiT – it gave us vision, best practice guidelines, tangible milestones and has empowered us to take on similar future projects” – Tim Vogt, Lead Developer at Into The Source.
Since embedding DoiT Cloud Intelligence™, Into The Source has achieved a 27% reduction in cloud spending, improving predictability of unit economics. Although cost savings weren’t the initial focus, the platform has played a crucial role in optimizing expenses. The savings generated have been strategically reinvested into new projects and innovations, enabling the company to leverage these optimizations for further growth and development.
What's Next?
The data lake project is ongoing, and following this, DoiT will support Into The Source on its journey to secure ISO accreditation with clear roadmaps and governance guardrails. This will involve strengthening security posture and compliance on AWS, and prioritized roadmapping to determine the best strategy to achieve its ISO objectives.