Author Archive for: admin

Original Source: http://feedproxy.google.com/~r/1stwebdesigner/~3/44dpaH-PLD4/

Usually, the matter of security is something web developers are concerned with. Vulnerabilities can be caused by poorly written code, software not updated by users, something malicious injected into the site, or hackers finding a way to break through the main login area. However a website gets compromised, you don’t often hear stories about a design gone rogue. As such, web designers might not be too concerned with security or what role their design elements play within it.

That said, there are certain design elements that could impact your site’s security. Understanding how those elements may be compromised and how they affect your site will help you make smarter choices when it comes to using them. It’s simply a matter of knowing what they are, so you can be on the lookout.

If it’s not a part of your process yet, then I’m going to suggest that web security become something you pay closer attention to. Not sure exactly how or why that should be? Let’s take a look at the 8 design elements that have the potential to affect your site’s security.

1. Compromised Themes

Whenever you use a third-party software in order to build or start a design, be extra careful in which ones you choose. Even if you download or purchase a theme from a trusted repository, you could still run into trouble. Their software is just as susceptible to a security breach as any other piece of software.

Unfortunately, it’s not always the theme that’s been compromised either. Take the TimThumb exploit, for example. An image resizing tool called TimThumb was included in a number of WordPress themes, which opened any user of that theme to attack.

2. Compromised Plugins

Third-party extensions are another tool designers often use to create advanced or complicated design elements on websites. Again, because software is particularly vulnerable to hackers, you have to be very careful in which ones you use and how you maintain them.

The image slider plugin, Slider Revolution (RevSlider), was one such case of an exploit introduced to websites by a third-party extension.

3. Compromised Components

For those of you who code design elements from-scratch, you’re not necessarily any safer than plugin and theme users are from security breaches. Especially if you utilize pre-written components like CSS or JavaScript snippets, libraries, or frameworks.

That’s not to say you can’t trust code from others, but you should, again, be mindful of the quality of component you hook into your design.

4. Malicious File Uploads

In order for a website to be successful in converting visitors to paying customers or subscribed members, there needs to be a way for it to actively engage with them. Contact forms and comments systems can be particularly troublesome when they’re not protected against spam. However, there are also design elements that can come from external users that cause issues with security.

Specifically, if your website accepts file uploads from users–for instance, if users upload visuals to guest-submitted content or submit images for a contest–you could be putting your site at risk. That doesn’t mean you shouldn’t accept media from other users; it just means having your developer implement ways to better vet those files and ensure it’s not a way to inject malicious code.

5. Malvertising

There are a number of ways to monetize a website. But if you want to do so with an element that more seamlessly integrates with your design, you would probably use a system like AdSense that places ad content on your site. Again, though, you have to trust that the third party behind the ad content doesn’t mean your site or its visitors any harm.

In 2015, AdSense made the news when malvertising campaigns were discovered on users’ websites. These ads seemed harmless in nature until visitors started being redirected to scam sites.

6. Phishing Pages

When you design a site, you’re very careful about including pages that are necessary for the user’s experience. Every now and again, you may build a landing page that exists outside of the navigation, but is there for the purposes of promoting something special. This is a commonly used marketing tactic.

However, hackers are well-aware of this, too, which is why some of them are able to get away with planting fake pages on legit websites. This is what’s known as a phishing page and it’s built with the purposes of installing malware on a visitor’s computer or directing them to a scam site.

7. Infected Images

Did you know that malware can be hidden inside of images, too? In 2011, this very problem was discovered within Google Images. In 2015, Saumil Shah gave a presentation at the HITB Security Conference where he discussed the Stegosploit hack.

Both of these examples demonstrate how hackers have learned to get around users’ growing awareness of malicious links and attachments. Now, they’ve found a way to compromise images.

8. Mixed Content

As the web moves towards a more secure place with the adoption of SSL certificates and HTTPS, some web designers run into trouble when media files aren’t properly shifted over to a secure address. This is what’s known as “mixed content”.

Basically, this is what happens when a website resides on a secure HTTP domain (HTTPS). However, when images remain sitting on the unsecured server (HTTP), they open the rest of the website up to the possibility of a breach.

Protect Your Design Elements

You’ve built a stunning design for your client, but worry now that your choice of design elements or third-party integrations may compromise the security of the site. While the probability of something like that happening might not be high, you don’t want to leave your design assets unprotected.

If you and your web developer haven’t done so yet, focus on setting up the following on your web server:

SSL certificate
Firewall
Malware and DDoS protection
Brute force protection
Scheduled software updates

And if you’re nervous about being able to secure these elements–as well as the rest of the site–on your own, then it’s time you considered moving to secure WordPress hosting. A managed WordPress hosting provider will take care of security, so you can focus strictly on the design piece of your clients’ websites.

Original Source: https://inspiredm.com/exclusive-freebie-start-up-icons/

Every day there’s a new opportunity, a new venue & new horizons for the entrepreneurs with the aim to make it happen, that’s why we’re looking to help: check out the latest free icons pack from Iconshock.com & bypeople.com: 30 unique items, including vector Adobe Illustrator & SVG Files, also adding PNG’s in three different sizes.

All the icons in this pack are free for both personal and commercial projects, so you won’t find any limits to focus on the important: make your business profitable and sustainable!

Get these amazing icons from here

The post Exclusive Freebie: Start Up Icons appeared first on Inspired Magazine.

Original Source: https://www.smashingmagazine.com/2018/05/desktop-wallpaper-calendars-june-2018/

Cheerful Desktop Wallpapers To Kick Off June (2018 Edition)

Cheerful Desktop Wallpapers To Kick Off June (2018 Edition)

Cosima Mielke

2018-05-31T08:43:08+02:00
2018-05-31T11:49:45+00:00

We all need a little inspiration boost every once in a while. And, well, whatever your strategy to get your creative juices flowing might be, sometimes inspiration lies closer than you think. As close as your desktop even.

Since more than nine years, we’ve been asking the design community to create monthly desktop wallpaper calendars. Wallpapers that are a bit more distinctive as what you’ll usually find out there, bound to cater for a little in-between inspiration spark. Of course, it wasn’t any different this time around.

This post features wallpapers for June 2018. All of them come in versions with and without a calendar, so it’s up to you to decide if you want to have the month at a glance or keep things simple. As a bonus goodie, we also collected some timeless June favorites from past years for this edition (please note that they thus don’t come with a calendar). A big thank-you to all the artists who have submitted their wallpapers and are still diligently continuing to do so. It’s time to freshen up your desktop!

Please note that:

All images can be clicked on and lead to the preview of the wallpaper,
You can feature your work in our magazine by taking part in our Desktop Wallpaper Calendar series. We are regularly looking for creative designers and artists to be featured on Smashing Magazine. Are you one of them?

What if there was a web conference without… slides? Meet SmashingConf Toronto 2018 ?? with live sessions exploring how experts work behind the scenes. Dan Mall, Lea Verou, Sara Soueidan, Seb Lee-Delisle and many others. June 26–27. With everything from live designing to live performance audits.

Check the speakers →

Travel Time

“June is our favorite time of the year because the keenly anticipated sunny weather inspires us to travel. Stuck at the airport, waiting for our flight but still excited about wayfaring, we often start dreaming about the new places we are going to visit. Where will you travel to this summer? Wherever you go, we wish you a pleasant journey!” — Designed by PopArt Studio from Serbia.

preview
with calendar: 320×480, 640×480, 800×480, 800×600, 1024×768, 1024×1024, 1152×864, 1280×720, 1280×800, 1280×960, 1280×1024, 1400×1050, 1440×900, 1600×1200, 1680×1050, 1680×1200, 1920×1080, 1920×1200, 1920×1440, 2560×1440
without calendar: 320×480, 640×480, 800×480, 800×600, 1024×768, 1024×1024, 1152×864, 1280×720, 1280×800, 1280×960, 1280×1024, 1400×1050, 1440×900, 1600×1200, 1680×1050, 1680×1200, 1920×1080, 1920×1200, 1920×1440, 2560×1440

Tropical Vibes

“With summer just around the corner, I’m already feeling the tropical vibes.” — Designed by Melissa Bogemans from Belgium.

preview
with calendar: 320×480, 640×480, 800×480, 800×600, 1024×768, 1024×1024, 1152×864, 1280×720, 1280×800, 1280×960, 1280×1024, 1400×1050, 1440×900, 1600×1200, 1680×1050, 1680×1200, 1920×1080, 1920×1200, 1920×1440, 2560×1440
without calendar: 320×480, 640×480, 800×480, 800×600, 1024×768, 1024×1024, 1152×864, 1280×720, 1280×800, 1280×960, 1280×1024, 1400×1050, 1440×900, 1600×1200, 1680×1050, 1680×1200, 1920×1080, 1920×1200, 1920×1440, 2560×1440

No Matter What, I’ll Be There

“June is the month when we celebrate our dads and to me the best ones are the ones that even if they don’t like what you are doing they’ll be by your side.” — Designed by Maria Keller from Mexico.

preview
with calendar: 320×480, 640×480, 640×1136, 750×1334, 800×480, 800×600, 1024×768, 1024×1024, 1125×2436, 1152×864, 1242×2208, 1280×720, 1280×800, 1280×960, 1280×1024, 1366×768, 1400×1050, 1440×900, 1600×1200, 1680×1050, 1680×1200, 1920×1080, 1920×1200, 1920×1440, 2560×1440, 2880×1800
without calendar: 320×480, 640×480, 640×1136, 750×1334, 800×480, 800×600, 1024×768, 1024×1024, 1125×2436, 1152×864, 1242×2208, 1280×720, 1280×800, 1280×960, 1280×1024, 1366×768, 1400×1050, 1440×900, 1600×1200, 1680×1050, 1680×1200, 1920×1080, 1920×1200, 1920×1440, 2560×1440, 2880×1800

We Sailed Across The Summer Sea

“Summer is the season of family trips, outings and holidays. What better way is there to celeberate Father’s Day, than to enjoy the summer sailing across with father beloved.” — Designed by The Whisky Corporation from Singapore.

preview
with calendar: 320×480, 640×480, 800×480, 800×600, 1024×768, 1024×1024, 1152×864, 1280×720, 1280×800, 1280×960, 1280×1024, 1366×768, 1400×1050, 1440×900, 1600×1200, 1680×1050, 1680×1200, 1920×1080, 1920×1200, 1920×1440, 2560×1440
without calendar: 320×480, 640×480, 800×480, 800×600, 1024×768, 1024×1024, 1152×864, 1280×720, 1280×800, 1280×960, 1280×1024, 1366×768, 1400×1050, 1440×900, 1600×1200, 1680×1050, 1680×1200, 1920×1080, 1920×1200, 1920×1440, 2560×1440

Sounds Of Rain

“When it rains the world softens and listens to the gentle pattering of the rain. It gets covered in a green vegetation and puddles reflect the colors of nature now looking fresh and cleansed. The lullaby of the rain is accustomed to creaky frogs and soft chirping birds taking you to ecstasy.” — Designed by Aviv Digital from India.

preview
with calendar: 320×480, 640×480, 800×480, 800×600, 1024×768, 1024×1024, 1152×864, 1280×720, 1280×800, 1280×960, 1280×1024, 1366×768, 1400×1050, 1440×900, 1600×1200, 1680×1050, 1680×1200, 1920×1080, 1920×1200, 1920×1440, 2560×1440
without calendar: 320×480, 640×480, 800×480, 800×600, 1024×768, 1024×1024, 1152×864, 1280×720, 1280×800, 1280×960, 1280×1024, 1366×768, 1400×1050, 1440×900, 1600×1200, 1680×1050, 1680×1200, 1920×1080, 1920×1200, 1920×1440, 2560×1440

-SmashingConf Toronto

“What’s the best way to learn? By observing designers and developers working live. For our new conference in Toronto, all speakers aren’t allowed to use any slides at all. Welcome SmashingConf #noslides, a brand new conference in Toronto, full of interactive live sessions, showing how web designers design and how web developers build — including setup, workflow, design thinking, naming conventions and everything in-between.” — Designed by Ricardo Gimenes from Brazil.

preview
with calendar: 320×480, 640×480, 800×480, 800×600, 1024×768, 1024×1024, 1152×864, 1280×720, 1280×800, 1280×960, 1280×1024, 1366×768, 1400×1050, 1440×900, 1600×1200, 1680×1050, 1680×1200, 1920×1080, 1920×1200, 1920×1440, 2560×1440
without calendar: 320×480, 640×480, 800×480, 800×600, 1024×768, 1024×1024, 1152×864, 1280×720, 1280×800, 1280×960, 1280×1024, 1366×768, 1400×1050, 1440×900, 1600×1200, 1680×1050, 1680×1200, 1920×1080, 1920×1200, 1920×1440, 2560×1440

Are You Ready?

“The excitement is building, the slogans are ready, the roaring and the cheering make their way… Russia is all set for the football showdown. Are you ready?” — Designed by Sweans from London.

preview
with calendar: 320×480, 640×480, 800×480, 800×600, 1024×768, 1152×864, 1280×720, 1280×800, 1280×960, 1280×1024, 1366×768, 1400×1050, 1440×900, 1600×1200, 1680×1050, 1680×1200, 1920×1080, 1920×1200, 1920×1440, 2560×1440
without calendar: 320×480, 640×480, 800×480, 800×600, 1024×768, 1152×864, 1280×720, 1280×800, 1280×960, 1280×1024, 1366×768, 1400×1050, 1440×900, 1600×1200, 1680×1050, 1680×1200, 1920×1080, 1920×1200, 1920×1440, 2560×1440

Summer Surf

“Summer vibes…” — Designed by Antun Hirsman from Croatia.

preview
with calendar: 640×480, 1152×864, 1280×1024, 1440×900, 1680×1050, 1920×1080, 1920×1440, 2650×1440
without calendar: 640×480, 1152×864, 1280×1024, 1440×900, 1680×1050, 1920×1080, 1920×1440, 2650×1440

Stop Child Labor

“‘Child labor and poverty are inevitably bound together, and if you continue to use the labor of children as the treatment for the social disease of poverty, you will have both poverty and child labor to the end of time.’ (Grace Abbott)” — Designed by Dipanjan Karmakar from India.

preview
with calendar: 1280×800, 1280×960, 1280×1024, 1400×1050, 1440×900, 1680×1050, 1920×1080
without calendar: 1280×800, 1280×960, 1280×1024, 1400×1050, 1440×900, 1680×1050, 1920×1080

Is your pattern library up to date today? Alla Kholmatova has just finished a fully fledged book on Design Systems and how to get them right. With common traps, gotchas and the lessons she learned. Hardcover, eBook. Just sayin’.

Table of Contents →

June Best-Of

Some things are too good to be forgotten. That’s why we dug out some June favorites from our archives. Please note that these don’t come with a calendar because of this. Enjoy!

Oh, The Places You Will Go!

“In celebration of high school and college graduates ready to make their way in the world!” — Designed by Bri Loesch from the United States.

preview
without calendar: 320×480, 1024×768, 1280×1024, 1440×900, 1680×1050, 1680×1200, 1920×1440, 2560×1440

Join The Wave

“The month of warmth and nice weather is finally here. We found inspiration in the World Oceans Day which occurs on June 8th and celebrates the wave of change worldwide. Join the wave and dive in!” — Designed by PopArt Studio from Serbia.

preview
without calendar: 320×480, 640×480, 800×480, 800×600, 1024×768, 1024×1024, 1152×864, 1280×720, 1280×800, 1280×960, 1280×1024, 1366×768, 1400×1050, 1440×900, 1600×1200, 1680×1050, 1680×1200, 1920×1080, 1920×1200, 1920×1440, 2560×1440

Solstice Sunset

“June 21 marks the longest day of the year for the Northern Hemisphere – and sunsets like these will be getting earlier and earlier after that!” — Designed by James Mitchell from the United Kingdom.

preview
without calendar: 1280×720, 1280×800, 1366×768, 1440×900, 1680×1050, 1920×1080, 1920×1200, 2560×1440, 2880×1800

Midsummer Night’s Dream

“The summer solstice in the northern hemisphere is nigh. Every June 21 we celebrate the longest day of the year and, very often, end up dancing like pagans. Being landlocked, we here in Serbia can only dream about tidal waves and having fun at the beach. What will your Midsummer Night’s Dream be?” — Designed by PopArt Studio from Serbia.

preview
without calendar: 320×480, 640×480, 800×480, 800×600, 1024×768, 1024×1024, 1152×864, 1280×720, 1280×800, 1280×960, 1280×1024, 1366×768, 1440×900, 1440×1050, 1600×1200, 1680×1050, 1680×1200, 1920×1080, 1920×1200, 1920×1440, 2560×1440

Flamingood Vibes Only

“I love flamingos! They give me a happy feeling that I want to share with the world.” — Designed by Melissa Bogemans from Belgium.

preview
without calendar: 320×480, 640×480, 800×480, 800×600, 1024×768, 1024×1024, 1152×864, 1280×720, 1280×800, 1280×960, 1280×1024, 1400×1050, 1440×900, 1600×1200, 1680×1050, 1680×1200, 1920×1200, 1920×1440, 2560×1440

Papa Merman

“Dream away for a little while to a land where June never ends. Imagine the ocean, feel the joy of a happy and carefree life with a scent of shrimps and a sound of waves all year round. Welcome to the world of Papa Merman!” — Designed by GraphicMama from Bulgaria.

preview
without calendar: 320×480, 640×480, 800×480, 800×600, 1024×768, 1024×1024, 1152×864, 1280×720, 1280×800, 1280×960, 1280×1024, 1366×768, 1400×1050, 1440×900, 1600×1200, 1680×1050, 1680×1200, 1920×1080, 1920×1200, 1920×1440, 2560×1440

Strawberry Fields

Designed by Nathalie Ouederni from France.

preview
without calendar: 320×480, 1024×768, 1280×1024, 1440×900, 1680×1200, 1920×1200, 2560×1440

Fishing Is My Passion!

“The month of June is a wonderful time to go fishing, the most soothing and peaceful activity.” — Designed by Igor Izhik from Canada.

preview
without calendar: 320×480, 640×480, 800×480, 800×600, 1024×768, 1024×1024, 1152×864, 1280×720, 1280×800, 1280×960, 1280×1024, 1400×1050, 1440×900, 1600×1200, 1680×1200, 1920×1080, 1920×1200, 1920×1440, 2560×1440

Summer Time

“Summer is coming so I made this simple pattern with all my favorite summer things.” — Designed by Maria Keller from Mexico.

preview
without calendar: 320×480, 640×480, 640×1136, 750×1334, 800×480, 800×600, 1024×768, 1024×1024, 1152×864, 1242×2208, 1280×720, 1280×800, 1280×960, 1280×1024, 1366×768, 1400×1050, 1440×900, 1600×1200, 1680×1050, 1680×1200, 1920×1080, 1920×1200, 1920×1440, 2560×1440, 2880×1800

The Amazing Water Park

“Summer is coming. And it’s going to be like an amazing water park, full of stunning encounters.” — Designed by Netzbewegung / Mario Metzger from Germany.

preview
without calendar: 320×480, 640×480, 800×600, 1024×768, 1024×1024, 1152×864, 1280×720, 1280×800, 1280×960, 1280×1024, 1366×768, 1400×1050, 1440×900, 1600×1200, 1680×1050, 1680×1200, 1920×1080, 1920×1200, 1920×1440, 2560×1440

Gravity

Designed by Elise Vanoorbeek (Doud Design) from Belgium.

preview
without calendar: 1024×768, 1024×1024, 1152×864, 1280×720, 1280×800, 1280×960, 1280×1024, 1366×768, 1400×1050, 1440×900, 1600×1200, 1680×1050, 1680×1200, 1920×1080, 1920×1200, 1920×1440, 2560×1440

Periodic Table Of HTML5 Elements

“We wanted an easy reference guide to help navigate through HTML5 and that could be updateable” — Designed by Castus from the UK.

preview
without calendar: 1024×768, 1280×800, 1280×1024, 1366×768, 1440×900, 1600×900, 1600×1200, 1920×1080, 1920×1200, 2560×1440

Ice Creams Away!

“Summer is taking off with some magical ice cream hot air balloons.” — Designed by Sasha Endoh from Canada

preview
without calendar: 320×480, 1024×768, 1152×864, 1280×800, 1280×960, 1400×1050, 1440×900, 1600×1200, 1680×1050, 1920×1080, 1920×1200, 2560×1440

Lavender Is In The Air!

“June always reminds me of lavender — it just smells wonderful and fresh. For this wallpaper I wanted to create a simple, yet functional design that featured… you guessed it… lavender!” — Designed by Jon Phillips from Canada.

preview
without calendar: 320×480, 640×480, 800×480, 800×600, 1024×768, 1024×1024, 1152×864, 1280×720, 1280×800, 1280×960, 1280×1024, 1400×1050, 1440×900, 1600×1200, 1680×1050, 1680×1200, 1920×1080, 1920×1200, 1920×1440, 2560×1440

Ice Cream June

“For me, June always marks the beginning of summer! The best way to celebrate summer is of course ice cream, what else?” — Designed by Tatiana Anagnostaki from Greece.

preview
without calendar: 1024×768, 1280×1024, 1440×900, 1680×1050, 1680×1200, 1920×1440, 2560×1440

Join In Next Month!

Please note that we respect and carefully consider the ideas and motivation behind each and every artist’s work. This is why we give all artists the full freedom to explore their creativity and express emotions and experience throughout their works. This is also why the themes of the wallpapers weren’t anyhow influenced by us, but rather designed from scratch by the artists themselves.

Thank you to all designers for their participation. Join in next month!

Original Source: https://www.smashingmagazine.com/2018/05/building-central-logging-service/

Building A Central Logging Service In-House

Building A Central Logging Service In-House

Akhil Labudubariki

2018-05-30T13:30:22+02:00
2018-05-30T12:20:54+00:00

We all know how important debugging is for improving application performance and features. BrowserStack runs one million sessions a day on a highly distributed application stack! Each involves several moving parts, as a client’s single session can span multiple components across several geographic regions.

Without the right framework and tools, the debugging process can be a nightmare. In our case, we needed a way to collect events happening during different stages of each process in order to get an in-depth understanding of everything taking place during a session. With our infrastructure, solving this problem became complicated as each component might have multiple events from their lifecycle of processing a request.

That’s why we developed our own in-house Central Logging Service tool (CLS) to record all important events logged during a session. These events help our developers identify conditions where something goes wrong in a session and helps keep track of certain key product metrics.

Debugging data ranges from simple things like API response latency to monitoring a user’s network health. In this article, we share our story of building our CLS tool which collects 70G of relevant chronological data per day from 100+ components reliably, at scale and with two M3.large EC2 instances.

Getting the process just right ain’t an easy task. That’s why we’ve set up ‘this-is-how-I-work’-sessions — with smart cookies sharing what works really well for them. A part of the Smashing Membership, of course.

Explore features →

The Decision To Build In-House

First, let’s consider why we built our CLS tool in-house rather than used an existing solution. Each of our sessions sends 15 events on average, from multiple components to the service – translating into approximately 15 million total events per day.

Our service needed the ability to store all this data. We sought a complete solution to support event storing, sending and querying across events. As we considered third-party solutions such as Amplitude and Keen, our evaluation metrics included cost, performance in handling high parallel requests and ease of adoption. Unfortunately, we could not find a fit that met all our requirements within budget – although benefits would have included saving time and minimizing alerts. While it would take additional effort, we decided to develop an in-house solution ourselves.

One of the biggest issues with building In-house is the amount of resources that we need to spend to maintain it. (Image credit: Source: Digiday)

Technical Details

In terms of architecting for our component, we outlined the following basic requirements:

Client Performance
Does not impact the performance of the client/component sending the events.
Scale
Able to handle a high number of requests in parallel.
Service performance
Quick to process all events being sent to it.
Insight into data
Each event logged needs to have some meta information to be able to uniquely identify the component or user, account or message and give more information to help the developer debug faster.
Queryable interface
Developers can query all events for a particular session, helping to debug a particular session, build component health reports, or generate meaningful performance statistics of our systems.
Faster and easier adoption
Easy integration with an existing or new component without burdening teams and taking up their resources.
Low maintenance
We are a small engineering team, so we sought a solution to minimize alerts!

Building Our CLS Solution

Decision 1: Choosing An Interface To Expose

In developing CLS, we obviously didn’t want to lose any of our data, but we didn’t want component performance to take a hit either. Not to mention the additional factor of preventing existing components from becoming more complicated, since it would delay overall adoption and release. In determining our interface, we considered the following choices:

Storing events in local Redis in each component, as a background processor pushes it to CLS. However, this requires a change in all components, along with an introduction of Redis for components which didn’t already contain it.
A Publisher – Subscriber model, where Redis is closer to the CLS. As everyone publishes events, again we have the factor of components running across the globe. During the time of high-traffic, this would delay components. Further, this write could intermittently jump up to five seconds (due to the internet alone).
Sending events over UDP, which offers a lesser impact on application performance. In this case data would be sent and forgotten, however, the disadvantage here would be data loss.

Interestingly, our data loss over UDP was less than 0.1 percent, which was an acceptable amount for us to consider building such a service. We were able to convince all teams that this amount of loss was worth the performance, and went ahead to leverage a UDP interface that listened to all events being sent.

While one result was a smaller impact on an application’s performance, we did face an issue as UDP traffic was not allowed from all networks, mostly from our users’ – causing us in some cases to receive no data at all. As a workaround, we supported logging events using HTTP requests. All events coming from the user’s side would be sent via HTTP, whereas all events being recorded from our components would be via UDP.

Decision 2: Tech Stack (Language, Framework & Storage)

We are a Ruby shop. However, we were uncertain if Ruby would be a better choice for our particular problem. Our service would have to handle a lot of incoming requests, as well as process a lot of writes. With the Global Interpreter lock, achieving multithreading or concurrency would be difficult in Ruby (please don’t take offense – we love Ruby!). So we needed a solution that would help us achieve this kind of concurrency.

We were also keen to evaluate a new language in our tech stack, and this project seemed perfect for experimenting with new things. That’s when we decided to give Golang a shot since it offered inbuilt support for concurrency and lightweight threads and go-routines. Each logged data point resembles a key-value pair where ‘key’ is the event and ‘value’ serves as its associated value.

But having a simple key and value is not enough to retrieve a session related data – there is more metadata to it. To address this, we decided any event needing to be logged would have a session ID along with its key and value. We also added extra fields like timestamp, user ID and the component logging the data, so that it became more easy to fetch and analyze data.

Now that we decided on our payload structure, we had to choose our datastore. We considered Elastic Search, but we also wanted to support update requests for keys. This would trigger the entire document to be re-indexed, which might affect the performance of our writes. MongoDB made more sense as a datastore since it would be easier to query all events based on any of the data fields that would be added. This was easy!

Decision 3: DB Size Is Huge And Query And Archiving Sucks!

In order to cut maintenance, our service would have to handle as many events as possible. Given the rate that BrowserStack releases features and products, we were certain the number of our events would increase at higher rates over time, meaning our service would have to continue to perform well. As space increases, reads and writes take more time – which could be a huge hit on the service’s performance.

The first solution we explored was moving logs from a certain period away from the database (in our case, we decided on 15 days). To do this, we created a different database for each day, allowing us to find logs older than a particular period without having to scan all written documents. Now we continually remove databases older than 15 days from Mongo, while of course keeping backups just in case.

The only leftover piece was a developer interface to query session-related data. Honestly, this was the easiest problem to solve. We provide an HTTP interface, where people can query for session related events in the corresponding database in the MongoDB, for any data having a particular session ID.

Architecture

Let’s talk about the internal components of the service, considering the following points:

As previously discussed, we needed two interfaces – one listening over UDP and another listening over HTTP. So we built two servers, again one for each interface, to listen for events. As soon as an event arrives, we parse it to check whether it has the required fields – these are session ID, key, and value. If it does not, the data is dropped. Otherwise, the data is passed over a Go channel to another goroutine, whose sole responsibility is to write to the MongoDB.
A possible concern here is writing to the MongoDB. If writes to the MongoDB are slower than the rate data is received, this creates a bottleneck. This, in turn, starves other incoming events and means dropped data. The server, therefore, should be fast in processing incoming logs and be ready to process ones upcoming. To address the issue, we split the server into two parts: the first receives all events and queues them up for the second, which processes and writes them into the MongoDB.
For queuing we chose Redis. By dividing the entire component into these two pieces we reduced the server’s workload, giving it room to handle more logs.
We wrote a small service using Sinatra server to handle all the work of querying MongoDB with given parameters. It returns an HTML/JSON response to developers when they need information on a particular session.

All these processes happily run on a single m3.large instance.

CLS v1: A representation of the system’s first architecture. All the components are running on one single machine.

Feature Requests

As our CLS tool saw more use over time, it needed more features. Below, we discuss these and how they were added.

Missing Metadata

Gradually as the number of components in BrowserStack increases, we’ve demanded more from CLS. For example, we needed the ability to log events from components lacking a session ID. Otherwise obtaining one would burden our infrastructure, in the form of affecting application performance and incurring traffic on our main servers.

We addressed this by enabling event logging using other keys, such as terminal and user IDs. Now whenever a session is created or updated, CLS is informed with the session ID, as well as the respective user and terminal IDs. It stores a map that can be retrieved by the process of writing to MongoDB. Whenever an event that contains either the user or terminal ID is retrieved, the session ID is added.

Handle Spamming (Code Issues In Other Components)

CLS also faced the usual difficulties with handling spam events. We often found deploys in components that generated a huge volume of requests sent to CLS. Other logs would suffer in the process, as the server became too busy to process these and important logs were dropped.

For the most part, most of the data being logged were via HTTP requests. To control them we enable rate limiting on nginx (using the limit_req_zone module), which blocks requests from any IP we found hitting requests more than a certain number in a small amount of time. Of course, we do leverage health reports on all blocked IPs and inform the responsible teams.

Scale v2

As our sessions per day increased, data being logged to CLS was also increasing. This affected the queries our developers were running daily, and soon the bottleneck we had was with the machine itself. Our setup consisted of two core machines running all of the above components, along with a bunch of scripts to query Mongo and keep track of key metrics for each product. Over time, data on the machine had increased heavily and scripts began to take a lot of CPU time. Even after trying to optimizing Mongo queries, we always came back to the same issues.

To solve this, we added another machine for running health report scripts and the interface to query these sessions. The process involved booting a new machine and setting up a slave of the Mongo running on the main machine. This has helped reduce the CPU spikes we saw every day caused by these scripts.

CLS v2: A representation of the current system’s architecture. Logs are written to the master machine and they are synced on the slave machine. Developer’s queries run on the slave machine.

Conclusion

Building a service for a task as simple as data logging can get complicated, as the amount of data increases. This article discusses the solutions we explored, along with challenges faced while solving this problem. We experimented with Golang to see how well it would fit with our ecosystem, and so far we have been satisfied. Our choice to create an internal service rather than paying for an external one has been wonderfully cost-efficient. We also didn’t have to scale our setup to another machine until much later – when the volume of our sessions increased. Of course, our choices in developing CLS were completely based on our requirements and priorities.

Today CLS handles up to 15 million events every day, constituting up to 70 GB of data. This data is being used to help us solve any issues our customers face during any session. We also use this data for other purposes. Given the insights each session’s data provides on different products and internal components, we’ve begun leveraging this data to keep track of each product. This is achieved by extracting the key metrics for all the important components.

All in all, we’ve seen great success in building our own CLS tool. If it makes sense for you, I recommend you consider doing the same!

(rb, ra, il)