97 Things every Cloud Engineer should Know – Summarised Notes

stephen book reviews, Cloud, Server

As with the other post on 97 things every developer should know, this is not a hard and fast list.

It is a bit of interesting info from people in the industry that may or may not have the required experience and may be in a specific part of the business not relevant to you.

It is however always useful to read, learn and grow...so here is a summary of the book

What is the Cloud?

On-demand Compute and storage resources - a datacenter accesible over the internet

Essential characteristics of the cloud model:

  • On-demand self-service - no need to call, open a ticket or send email for something to be provisioned
  • Broad network accessible - accessible from a multitude of devices on different networks
  • Resource pooling - making a pool of resources available to multiple customers - not knowing where the physical machine is. An abstraction.
  • Rapid elasticity - resources should scale up and down based on customer requirement - an engineer need not worry about adding and removing capacity
  • Measurred service - pay for what you use for a given unit of time

Why the cloud?

Procuring new hardware, installing and maintaining takes a long time - taking away from the key to company revenues - the application

Technology can be a cost saver and an enabler...

Three keys to making the right multicloud decision

  • visibility - having information where it matters a trusted source of truth
  • efficiency - best use of resources
  • governance - security, quality and policy enforcement allows for improved innovation and speed of delivery

Use Managed Service - Please

Operations at scale are a hard problem - failure is rampant

Automation and standardisation is crucial

Managed services like Amazon Relational DB (RDS), google cloud SQL or Microsoft Azure SQL.

They have the experience and have been through the trouble. Unfortunately these are proprietary things and limit your freedom - so there is a tradeoff between using standard MySQL or Postgres and the big cloud version of the service.

Examples:

  • File storage
  • Object cache
  • Message Queues
  • Stream processing
  • ETL (Extract, Transform, Load)

Why?:

  • They have experience
  • Cheaper
  • Faster for development

Why not?:

  • You cannot tweak anything
  • Lock-in

A Cloud Computing Vocabulary

  • Availability - Ampunt of time (as a percentage) of a service being live and functional
  • Durability - Chance that data will be lost or corrupted
  • Consistency - when writing to a datastore the data read is the same value immediately
  • Elasticity - Dynamically matching resource requirements based on the load
  • Scalability - how elasticity is achieved...vertical (bigger hammer) or horizontally (more hammers)
  • Serverless - Running application code without managing infrastructure also known as faas function as a service
  • Fully managed - cloud resources extracted away so the cloud provider takes all responsibility from the software developer

Every Engineer Should be a Cloud Engineer

If you are lazy and don't want to re-invent the wheel.
Describe your problem and their will be a cloud service for it - the trick is making it fit together.

Tips:

  • Turn on billing alerts
  • Make use of free-tier
  • Use external documentation
  • Change one setting at a time
  • Few people understand IAM (Identity and Access Management)

Manage Up: Engage with Executives in the Cloud

  • Understand what executives need
  • Tell them how your proposal meets their needs in their language
  • Be trusted in a world of marketing buzzwords and high expectations
  • KNow the numbers
  • Know how executive performance in measured

Architecture

Whats Next for Containers?

Combining the best of Containers and VMs: MicroVMs and UniKernels

MicroVM's: VM's for specific use cases, can boot in milliseconds.
MicroVM runtimes launch the vm then launch the container in it.

Unikernels: lightweight, immutable OS - application is combined with OS drivers and libraries. Example: MirageOS

Performance and security benefits.
Downside is they are completely different from containers.

Containers aren't Magic

The OCI (Open Container Initiative) is a definition of a seperate place for applications to live.

Your CIO wants to Replatform only Once

Pitch why new infrastructure will increase velocity, security, reliability, future proofing and scalability.

Must be cloud agnostic.

Kubernetes is the ultimate PaaS builder.

My ranking of Star Wars Books

stephen Uncategorized

  1. Path of Destruction (Star Wars: Darth Bane, #1)

  2. Thrawn (Star Wars: Thrawn, #1)

  3. Rule of Two (Star Wars: Darth Bane, #2)

  4. Master and Apprentice

  5. Dynasty of Evil (Star Wars: Darth Bane, #3)

  6. Heir to the Empire (Star Wars: The Thrawn Trilogy, #1)

  7. Into the Void (Star Wars Legends)

  8. Annihilation (Star Wars: The Old Republic (Publication Order) #4)

  9. Revan (Star Wars: The Old Republic (Chronological Order) #1)

  10. Heir to the Jedi (Star Wars Disney Canon Novel)

  11. Dark Disciple (Star Wars Disney Canon Novel)

  12. Kenobi (Star Wars Legends)

  13. Darth Plagueis (Star Wars Legends)

  14. Lost Tribe of the Sith: The Collected Stories (Star Wars: Lost Tribe of the Sith)

  15. Dooku: Jedi Lost (Star Wars Disney Canon Novel)

Displaying a network chart graph diagram on a Django site

stephen django

There is often a case for improving the understanding of things with visualisations.
Python provides a number of libraries to create great visualisations - however they often are focused more on a data science approach - scripts and jupyter notebooks.

What we want is the visualisations to be easily accessible through a django website, that is what I will be showing in this post with a specific focus on network diagrams.

What Visualisation Package are we using for the Network Graph

I have tried to find a few packages for creating a network graph:

Compararison

I will be using graphviz and networkx for very simple rudementary network graphs. You can make them look good but it is strenuous effort.

I will let you try networkx with plotly and igraph on your own.

Graphviz

Graphviz was very simple output and input. Not too much fuss and can render in many formats. To create a simple graph displayed on the frontend as an SVG:

from graphviz import Graph

g = Graph(
    'G',
    format='svg',
    engine='twopi',
)

g.node('root', shape='rectangle', width='1.5')
g.node('red')
g.node('blue')

g.edge('root', 'red', label='to_red')
g.edge('root', 'blue', label='to_blue')

context_data['my_chart'] = g.pipe().decode('utf-8')

Display on frontend:

{{ my_chart | safe }}

The image output:

graphviz-simple-radial-django

NetworkX

NetworkX is not primarily a graph drawing package but basic drawing with Matplotlib as well as an interface to use the open source Graphviz software package are included.

The networkx plot is drawn using matplotlib (it can also use graphviz) to draw.

    import matplotlib.pyplot as plt
    import networkx as nx
    import io

    G = nx.Graph()

    # rectanle width 1.5
    G.add_node('root')
    G.add_node('red')
    G.add_node('blue')

    # label: to_red
    G.add_edge('root', 'red')
    # label: to_blue
    G.add_edge('root', 'blue')

    nx.draw(G)
    buf = io.BytesIO()
    plt.savefig(buf, format='svg', bbox_inches='tight')
    image_bytes = buf.getvalue().decode('utf-8')
    buf.close()
    plt.close()

    context_data['my_chart'] = image_bytes

Display on frontend:

{{ my_chart | safe }}

The image output:

networkx-simple-django-network

No annotations...

Make sure you close the plot otherwise it will cause issues, matplotlib is not thread safe...

Important:

  • nx.draw(G) - draws with no labels
  • nx.draw_networkx(G) - Draws with labels

networkx-with-labels

Sources