Week 26 / 2023

Networking: Day 1

• Subnetting:
• the process of dividing a larger network into smaller subnetworks, known as subnets.
• ICMP:
• Internet Control Message Protocol. It is a network layer protocol in the Internet Protocol Suite (TCP/IP) that is primarily used for diagnostic and error-reporting purposes in computer networking.
• ICMP is typically implemented in the operating system of network devices, such as routers and hosts, to send error messages and operational information about the network.
• usages:
• Echo Request and Echo Reply (Ping): ICMP includes the Echo Request and Echo Reply messages, which are commonly known as "ping." By sending an Echo Request message to a destination device, you can check its availability and measure the round-trip time for the reply. Ping is widely used for network troubleshooting and monitoring.
• Destination Unreachable: The Destination Unreachable message is generated by a router or a host to inform the source device that the destination is unreachable for some reason. For example, the destination host may be down or the router may not have a route to the destination network.
• Time Exceeded: The Time Exceeded message is generated by a router or a host to inform the source device that the packet has been discarded because its Time to Live (TTL) value has reached zero. The TTL value is decremented by one by each router that forwards the packet. If the TTL value reaches zero, the packet is discarded and the Time Exceeded message is sent back to the source device.
• Traceroute:
• is a network diagnostic tool used to trace the route that packets take from a source device to a destination device over an IP network.
• It provides valuable information about the path and the time it takes for packets to travel through different routers or hops in the network.
• When you run a traceroute command, the source device sends a series of packets with increasing Time to Live (TTL) values. The TTL value is initially set to 1 for the first packet, 2 for the second packet, and so on. The TTL value represents the maximum number of hops a packet can traverse before being discarded.
• As the packets move through the network, each router that receives a packet decrements the TTL value by one. If the TTL reaches zero, the router discards the packet and sends an ICMP Time Exceeded message back to the source device. This process allows the source device to identify the routers along the path.
• Traceroute displays the IP addresses or hostnames of the routers, along with the round-trip time (RTT) for each hop. The RTT is the time it takes for a packet to travel from the source device to a specific router and back. By analyzing the traceroute results, network administrators can identify the network path, measure latency, and pinpoint any connectivity or performance issues.
• traceroute www.example.com.
• traceroute -m 15 www.google.com: maximum number of hops.
• traceroute -I www.github.com: use ICMP echo requests instead of UDP datagrams.

Python: Chapter 2, An Array of Sequences

• .sort() vs sorted(): .sort() in-place sorting, sorted() returns a new list.
• An important python API convention: functions or methods that change an object in place should return None to make it clear to the caller that the receiver was changed, and no new object was created.
• drawback of this: Since the methods return None, it is not possible to cascade or chain multiple method calls together in a single line of code. Cascading method calls is a technique where the result of one method call is immediately used to invoke another method on the returned object. This style of coding is often referred to as a "fluent interface." all str methods return a new string object, so you can chain them together. s.upper().lower().capitalize().
• Iterable: An iterable is any object that can be looped over using a loop construct, such as a for loop. It represents a sequence of elements and can be thought of as a container of values. Iterables are defined by implementing the __iter__() method, which returns an iterator.
• iterator: An iterator is an object that represents a stream of data or elements. It provides a way to access the elements of an iterable one by one, without storing them all in memory. Iterators maintain internal state to keep track of the current position within the stream. They implement the __next__() method, which returns the next element in the sequence, and the __iter__() method, which returns the iterator object itself. Iterators raise a StopIteration exception when there are no more elements to iterate over.
• Generator: A generator is a special type of iterator that can be defined using a yield statement within a function. It allows you to define an iterable sequence without explicitly implementing the iterator protocol. Generators are more concise and efficient than creating custom iterators from scratch. They automatically handle the state management and StopIteration exception. You can iterate over a generator using a for loop or by calling the next() function.
• In summary, an iterable is a container-like object that can be looped over, an iterator is an object that provides access to elements of an iterable one by one, and a generator is a special type of iterator defined using the yield statement. Iterators implement the iterator protocol, and iterables implement the iterable protocol by returning an iterator. These concepts enable efficient and flexible handling of sequences of data in Python.
• By default, Python sorts strings lexicographically by character code. That means ASCII uppercase letters will come before lowercase letters, and non-ASCII characters are unlikely to be sorted in a sensible way.
• The bisect module in Python provides support for binary search and insertion into sorted lists. It offers efficient algorithms for finding insertion points for items in sorted sequences. The module includes two primary functions: bisect() and insort().
• lists are more versatile and suitable for storing collections of different types of data with dynamic size requirements. On the other hand, arrays are optimized for storing large homogeneous data sets and provide efficient memory usage and faster element access.
• When List is not the answer:
• t for my_collection, especially if it holds a large number of items. Sets are optimized for fast membership checking. They are also iterable, but they are not sequences because the ordering of set items is unspecified.
• If a list only contains numbers, an array.array is a more efficient replacement.
• If you create an array('b'), then each item will be stored in a single byte and interpreted as an integer.
• array.tofile() and array.fromfile() methods to save and load arrays from files. they're very fast.
• Byte order refers to the arrangement of bytes within a multi-byte data type, such as integers. In little-endian byte order, the least significant byte is stored first, while in big-endian byte order, the most significant byte is stored first. By using the s.byteswap() method, you can convert the byte order of an integer or byte string to ensure compatibility with different systems or protocols that use a different byte order. Note that the byte order swap operation is only meaningful for multi-byte data types, such as integers, and may not have any visible effect on single-byte data types or byte strings.
• As of Python 3.10, the array type does not have an in-place sort method like list.sort().
• To keep a sorted array sorted while adding items to it, use the bisect.insort function.
• Memoryview:
• The memoryview class provides a shared memory mapping protocol for objects that support the buffer protocol (?). It allows you to access the internal data of an object that supports the buffer protocol without copying. This is useful for accessing large data sets, such as images or audio files, without incurring the overhead of copying the entire data set.
• a shared-memory sequence type that lets you handle slices of arrays without copying bytes.
• The memoryview object acts as a "window" or a "view" into the memory of another object, exposing a portion or the entirety of its data. It provides a convenient way to access and manipulate the data as if it were a regular mutable sequence, without the need to create a new copy of the data.
• memoryview is particularly useful when working with libraries or modules that expect buffer-like objects. It allows you to pass memory views directly to functions and APIs that operate on buffer objects without the need for additional data conversion or copying.
• Note that memoryview does not own the underlying data. It provides a view into the data of another object, and any modifications made through the memoryview will be reflected in the original object.
• NumPy:
• For advanced array and matrix operations. and provides efficient element-wise operations.
• Deque:
• But inserting and removing from the head of a list (the 0-index end) is costly because the entire list must be shifted in memory.
• The class collections.deque is a thread-safe double-ended queue designed for fast inserting and removing from both ends. It is also the way to go if you need to keep a list of “last seen items” or something of that nature. a deque can be bounded maxlen so that it never grows beyond a certain length. This is useful if you need a queue or stack structure with a maximum size. rotate() method to rotate the deque in-place. A positive argument rotates the deque to the right, and a negative argument rotates it to the left.
• But there is a hidden cost: removing items from the middle of a deque is not as fast. It is really optimized for appending and popping from the ends.
• summary:
• Mastering the standard library sequence types is a prerequisite for writing concise, effective, and idiomatic Python code.
• Tuples in Python play two roles: as records with unnamed fields and as immutable lists.
• Flat Versus Container Sequences: Flat sequences hold items of a single type, whereas container sequences can hold items of arbitrary types.
• there are containers in Python that are not sequences, like dict and set.

Postman

• Postman is a collaboration platform for API development. Postman's features simplify each step of building an API and streamline collaboration so you can create better APIs—faster.
• enables you to add dynamic behavior to requests and collections.
• Postman Collections are a group of saved requests.
• Postman Flows are a group of saved requests that are linked together. a visual tool for creating API workflows.
• api-first development? API-first development is a design-first approach to building APIs. It involves designing the API before writing any code. This approach helps you to think about the API from the client's perspective and design it to meet the client's needs. It also helps you to identify and resolve any potential issues early in the development process.
• API Documentation: Postman provides a powerful API documentation feature that allows you to create beautiful, customizable, and interactive API documentation pages for your APIs. You can use the Postman API to programmatically create and update documentation for your APIs.
• what is API pagination and the standard pagination techniques used in the present day?
• the use of parameters, such as offset and limit or cursor-based tokens, to control the size and position of the data subset to be retrieved.
• determine the starting point and the number of records to include on each page.
• It minimizes the load on servers, network bandwidth, and client-side applications.
• Common API Pagination Techniques:
• Common API Pagination Techniques
• Offset and Limit Pagination
• Cursor-Based Pagination: a unique identifier or token to mark the position in the dataset /api/posts?cursor=eyJpZCI6MX0 (represents the identifier of the last fetched record.)
• Page-Based Pagination: /api/posts?page=1&limit=10 (page number and the number of records to include on each page.) along with metadata such as the total number of pages or total record count.
• Time-Based Pagination: /api/posts?before=2021-10-01T00:00:00Z&after=2021-09-01T00:00:00Z (before and after parameters to specify the start and end dates of the data subset to be retrieved.) where data has a temporal aspect. enables fetching data in chronological or reverse-chronological order
• Best Practices for API Pagination:
• Use a Common Naming Convention for Pagination Parameters
• Always include Pagination Metadata in API Responses
• Determine an Appropriate Page Size:
• Understand the Data Characteristics
• Consider Network Latency and Bandwidth
• Evaluate Performance Impact
• Provide Flexibility with Pagination Parameters
• Handle Edge Cases and Error Conditions:
• Out-of-Range Page Requests
• Invalid Pagination Parameters
• Handling Empty Result Sets
• Server Errors and Exception Handling
• Rate Limiting and Throttling
• Clear and Informative Error Messages
• Consistent Error Handling Approach
• Consider Caching Strategies
• Page-Level Caching
• Result Set Caching
• Time-Based Caching
• Conditional Caching
• Reverse Proxy Caching

ERPNEXT

• Employee Checkin:

Python

Httpx

• provides sync and async APIs, and support for both HTTP/1.1 and HTTP/2.
• httpcore - The underlying transport implementation for httpx.
• pip install httpx[http2] - To install the optional HTTP/2 support.
• r = httpx.get(url, params=, headers=) - The get() function is a convenience function that performs a GET request and returns a Response object.
• r.url - The URL that was fetched.
• r.text - The response body as a string.
• r.content - The response body as bytes.
• r.json() - The response body parsed as JSON.
• r2 = httpx.post(url, data=, files={"file-name": open("XD.txt", "rb")}, json=, content=) - The post() function is a convenience function that performs a POST request and returns a Response object.
• httpx.codes - A dictionary mapping HTTP status codes to their descriptions.
• We can raise an exception for any responses which are not a 2xx success code: r.raise_for_status() - This will raise an HTTPStatusError or None
• httpx.stream() - The stream() function is a convenience function that performs a GET request and returns a Response object. It is similar to get(), but it returns a StreamingResponse object instead of a Response object.
• What is the difference between get() and stream()? The get() function will download the entire response body into memory before returning the Response object. The stream() function will return a StreamingResponse object, which will download the response body in chunks as it is read.
• iter_bytes() - The iter_bytes() function returns an iterator that yields the response body in chunks as bytes.
• iter_lines() - The iter_lines() function returns an iterator that yields the response body in chunks as strings, one line at a time.
• iter_raw() - The iter_raw() function returns an iterator that yields the response body in chunks as bytes.
• iter_text() - The iter_text() function returns an iterator that yields the response body in chunks as strings.
• If you're using streaming responses in any of these ways then the response.content and response.text attributes will not be available, and will raise errors if accessed.
• httpx.Cookies() - The Cookies class is a dictionary-like object that provides access to the cookies returned by the server. .set() - The set() method sets a cookie value.
• follow_redirects - Whether to follow redirects. Defaults to True. The history property of the response can be used to inspect any followed redirects.
• Timeouts: The default timeout for network inactivity is five seconds. timeout=int|None - The timeout value in seconds, or None for no timeout.
• HTTPX supports Basic and Digest HTTP authentication.
• HTTPX will raise exceptions if an error occurs. RequestError and HTTPStatusError are the two exceptions that HTTPX will raise.
• There is also a base class HTTPError that includes both of these categories, and can be used to catch either failed requests, or 4xx and 5xx responses.
• Async is a concurrency model that is far more efficient than multi-threading, and can provide significant performance benefits and enable the use of long-lived network connections such as WebSockets.
• asyncio, or trio - The underlying event loop implementation for httpx. as the backend for socket operations and concurrency primitives.
• async with httpx.AsyncClient() as client: - The AsyncClient class is the asynchronous equivalent of the Client class. It provides an asynchronous context manager interface, and supports asynchronous requests.
• Why use Client? HTTPX has to establish a new connection for every single request (connections are not reused). This is inefficient, and can be slow. If you're making a lot of requests, you should use a Client instance instead. a Client instance uses HTTP connection pooling. This means that when you make several requests to the same host, the Client will reuse the underlying TCP connection, instead of recreating one for every single request.

Cryptography

• Hill Cipher:
• Both codes and ciphers are methods for transforming a given message—the plaintext—into a new form that is unintelligible to anyone who does not know the rule—the key—for doing the transformation or who, more significantly, does not know the secret rule—the inverse key—for reversing the transformation in order to recover the original plaintext.
• a substitution cipher, where each letter of the plaintext is replaced by another letter. The key is a 26-letter permutation, which is a one-to-one mapping of the 26 letters of the alphabet onto itself.
• a polygraphic cipher, where the plaintext is divided into groups of adjacent letters of the same fixed length n, and then each such group is transformed into a different group of n letters. If n is not too small, then such a polygraphic substitution can render letter frequency analysis useless.
• The first systematic yet simple polygraphic ciphers using more than two letters per group are the ones we shall study below—the Hill ciphers.
• our alphabet consists of the 26 upper-case letters of the English alphabet followed by the period (.), the question mark ( ?), and the blank space, in that order. In plaintext, the period and question mark have their usual meaning, and the blank space is employed, as usual, to separate words
• To understand Hill ciphers, you first have to understand “modular arithmetic”.
• An arbitrary Hill n-cipher has as its key a given n × n matrix whose entries are nonnegative integers from among 0, 1, . . . , m − 1, where m is the length of the alphabet.
• Digraphs: The plaintext is divided into groups of two letters each, called digraphs. The last digraph may contain only one letter, in which case it is padded with a blank space.
• What we do is simply “wrap around” the numbers so that they are always between 0 and m − 1. This is called “reducing modulo m” or “taking the remainder modulo m”.
• Definition 1. We are given an integer m > 1, called the modulus. Then we say that two integers a and b congruent to one another modulo m (congruent “mod m” for short), and we write a≡b (mod m), to mean that the difference a − b is an integral multiple of m. In other words, a ≡ b (mod m) when a = b + k · m for some integer k (positive, negative, or zero).
• Thus the idea of “sameness” expressed by equivalence mod m generalizes the usual idea of equality.
• For our purposes, what is most important is that equivalence modulo m preserves sums, that is, adding one pair of integers that are congruent modulo m to a second pair of integers gives a sum of the first pair that is congruent to the sum of the second pair: if a ≡ c (mod m) and b ≡ d (mod m), then a + b ≡ c + d (mod m)
• Similarly, equivalence modulo m preserves products, that is, multiplying one pair of integers that are congruent modulo m to a second pair of integers gives a product of the first pair that is congruent to the product of the second pair: if a ≡ c (mod m) and b ≡ d (mod m), then a · b ≡ c · d (mod m)
• As in the example just worked, we may use arithmetic modulo m whenever we are doing matrix addition and multiplication, and we can freely take residues modulo m any step along the way without affecting the values modulo m in the final matrix answer.
• Suppose we are given an alphabet of length m > 1 and an integer n > 1. Then a Hill n-cipher is given by an n-by-n matrix A with entries in Zm.
• The Encryption Algorithm:
  1. Separate the plaintext from left to right into some number k of groups (polygraphs) of n letters each. If you run out of letters when forming the final group, repeat the last plaintext letter as many times as needed to fill out that final group to n letters.
  2. Replace each letter by the corresponding number of its position (from 0 through m − 1) in the alphabet to get k groups of n integers each.
  3. Reshape each of the k groups of integers into an n-row column vector and in turn multiply A by each of those k column vectors modulo m.
  4. After arranging all k of the resulting product n-row column vectors in order into a single (k · n)-vector (with entries in Zm ), replace each of these k · n entries with the corresponding letter of the alphabet. The result is the ciphertext corresponding to the original plaintext.
• The n-row column vectors formed in step 3, above, to represent groups of plaintext letters are called plaintext vectors.
• Likewise, the n-row column vectors obtained by multiplying them by A modulo m in step 3 are called ciphertext vectors
• First, we group the plaintext into polygraphs of length len(A) (the number of columns of A). If the last polygraph is not full, we pad it with the letter X (or with the last letter).
• if a Hill cipher has key matrix A, then the inverse transforma- tion is the Hill cipher whose key matrix is A−1 (inverse A).
• keywords: digraph, pad-out, ciphertext vectors, plaintext vectors.