Programming

Python – List Comprehensions

/ chenweilun2014 / Leave a comment

Basically, it produces a list but only once. That’s why it comes under or is associated with functional programming.

What is functional programming? It refers to computer programming using a “pipeline of pure functions”. (Héla Ben Khalfallah. 12 Jan 2021. Functional Programming, Simplified. https://betterprogramming.pub/simplified-functionaldd-programming-fdc07b4b1084).

What are functions? A function is a block of code that performs a specific task. (Parewa Labs Pvt. Ltd. Python Functions. https://www.programiz.com/python-programming/function. It “only runs when it is called”, so in this sense it offers more control to the software programmer. Once it runs, it’s gone (see Rodrigo Girão Serrão. 18 Jun 2022. List comprehensions in functional programming. https://mathspp.com/blog/twitter-threads/list-comprehensions-in-functional-programming). That is why some suggest functional programming makes your code more secure.

You will find below a modified replica of Christian Mayer’s example from Python One-Liners Write Concise, Eloquent Python Like a Professional (2020). https://github.com/finxter/PythonOneLiners/blob/master/book/python_tricks/one_liner_06.py. 

# modified example: execute List Comprehension with Slicing

## Data (daily copper prices ($))
price = [[10.91, 9.8, 9.8, 9.4, 9.5, 9.72, 8.88],
         [11.52, 9.4, 9.4, 9.3, 9.2, 9.13, 8.88],
         [8.45, 7.9, 7.9, 8.1, 8.0, 8.0, 9.88],
         [8.16, 5.9, 4.8, 4.8, 4.7, 3.99, 9.99],
         [11.52, 9.4, 9.4, 9.3, 9.26, 9.13, 11.06]]

## creates smaller sample to run computer modelling with less time
## extracts every other datapoint from each sub-list
sample = [line[::2] for line in price]

## note that the output is a list! You find that it begins with a square bracket and ends with a square bracket. 
print(sample)

[[10.91, 9.8, 9.5, 8.88], [11.52, 9.4, 9.2, 8.88], [8.45, 7.9, 8.0, 9.88], [8.16, 4.8, 4.7, 9.99], [11.52, 9.4, 9.26, 11.06]]

Hope it’s been fun for you!

Excel Helplines – {VBA + Excel}

/ chenweilun2014 / Leave a comment

Tips below, both VBA (programming) and Excel (alone):

  • [VBA – physically thick and heavy but likely to benefit beginners over long run] Brian D. Bissett. 2021. Automated Data Analysis Using Excel. Chapman and Hall/CRC. New York, US.
  • [VBA – physically small; useful for beginning to intermediate learners] Julitta Korol. 2016. Microsoft Excel 2016 programming: Pocket Primer. Dulles, Virginia : Mercury Learning and Information.
  • [VBA + Excel] Jordan Goldmeier. 2014. Advanced Excel essentials. Apress. New York, US.
  • [physically thick but light] Michael Alexander, Dick Kusleika. Excel 2019 power programming with VBA. [companion site -> https://www.wiley.com/en-us/Excel+2019+Power+Programming+with+VBA-p-9781119514923]
  • [accessible but basic introduction] Mike McGarth. 2019. Excel VBA in easy steps. In Easy Steps Ltd. Leamington Spa, Warwickshire, UK.
  • [VBA + Excel] https://chandoo.org/ [Purna Duggirala, currently in New Zealand]; recommended by Goldmeier (p. 8), Alexander and Kusleika (p.58).
  • [More Excel than VBA] https://www.contextures.com/index.html. [Debra Dalgleish. Toronto, Canada]; recommended by Goldmeier (p. 8), Alexander and Kusleika (p.58).
  • [VBA + Excel] Bill Jelen. https://www.youtube.com/c/MrExcelcom/featured. His forum was https://www.mrexcel.com/ was recommended by Alexander and Kusleika (p.58).
  • [VBA; mostly basic] https://www.geeksforgeeks.org/tag/excel-vba/. GeeksforGeeks. A-143, 9th Floor, Sovereign Corporate Tower, Sector-136, Noida, Uttar Pradesh – 201305. India.
  • [VBA + Excel, and others!] Learnit Training. https://www.youtube.com/c/LearnitTraining/playlists.

Associated

Python Project – Simple One

/ chenweilun2014 / Leave a comment

* Dear reader, you may consider re-sizing screen smaller than 90%; potentially removing the need to scroll right to view the code (I’ve re-written it with more vertical lines to make it more visual-friendly).

# Inspired by Andy Sterkowitz (GitHub post, 3 Levels of Portfolio Projects. https://github.com/andysterks/three-levels-of-projects; 3 Types of Projects That Will Make You a Programmer. https://www.youtube.com/watch?v=RYE0QQKJI9o).

# Reinforcing/refining Python fundamentals and problem-solving

# KIV enhance speed by replacing <if> + <elif> + <else> loops with hash/lookup tables e.g. dictionaries or tuples (see Nicholas Obert. Nov 1, 2021. https://betterprogramming.pub/get-rid-of-excessive-if-else-statements-with-lookup-and-hash-tables-c7dbed808996; Parewa Labs Pvt Ltd. accessed 1 Aug 2022. Python Lists Vs Tuples. https://www.programiz.com/python-programming/list-vs-tuples).

# Algorithm (problem-solving recipe/logic) uses: risk classification, test grading, commission amounts etc.

***

Given the below, create a programme to provide estimated license fees.

V. Anton Spraul. 2012. Think like a programmer : an introduction to creative problem solving. San Franciso. No Starch Press. p. 68 (book teaches thru C++)
def biz_license_cost():
    
    # prints only once
    print("Welcome to your yearly",end=" ") # same line
    print("business licence cost estimator!")
    print() # prettier layout
    
    '''
    Extended comments
    Completed - 2022 Jul 29
    
    Lines 40 to 46 
    <while True> + <try> + <except> + <else> 
    with <float>, for decimals:
    validates user input without crashing. 
    
    Lines multiple <break> 
    ensures the loop ends and exits the programme. 
    
    Line 59 <else> 
    resolves negative number input. 
    
    Line 64 <else> 
    covers final scenario for sales_amount > 500000, 
    and provides licence cost.
    
    Note to self
    Ensure alignment: <try> + <except> + <else>.
    <finally>, unused, is only other 
    part of this Exception process. 
    
    Further study
    Al Sweigart. 2019. accessed 2022 Jul 29. 
    San Francisco. No Starch Press. 
    Automate the Boring Stuff with Python, 2nd Edition. 
    Chapter 8. 
    https://automatetheboringstuff.com/2e/chapter8/. 
    '''
    
    while True: 
        try:
            sales_amount = float(input("Enter positive sales_amount $:"))
        except:
            print("Do only enter +ve",end=" ")
            print("amount without symbols")
        else:
            if 0 < sales_amount <= 50000:
                print("License -> $25",end=" ")
                print("/ Cat One")
                break
            elif (50000 < sales_amount <= 150000):
                print("License -> $200",end=" ")
                print("/ Cat Two")
                break
            elif (150000 < sales_amount <= 500000):
                print("License -> $1000",end=" ")
                print("/ Cat Three")
                break
            else:
                if int(sales_amount) < 0:
                    print("License -> None:",end=" ")
                    print("please reconsider your business options.")
                    break
                else:
                    print("License -> $5000",end=" ")
                    print("/ Cat Four")
                    break
        
biz_license_cost() # run function
Sample output = )

Python coding -> Excel

/ chenweilun2014 / 1 Comment

# to up office automation / efficiency

PIP -> Python programmers!

/ chenweilun2014 / Leave a comment

# Upgrading packages/libraries/modules generally reduce cyberattack risks and enhance performance;

# but they can break your code, so from the beginning it would be ideal to set up virtual environments for each coding project.

# If you use Python through data science platforms like Anaconda, there may be other things you have to type to upgrade.

# All the best!

***

“PIP is a package manager for Python packages, or modules…” (W3Schools. https://www.w3schools.com/python/python_pip.asp).

PCDIT -> Solve computing problems

/ chenweilun2014 / Leave a comment

This post came inspired by the reading of:

  • KURNIAWAN, Oka; JEGOUREL, Cyrille; LEE, Norman Tiong Seng; DE MARI, Matthieu; and POSKITT, Christopher M. Steps before syntax: Helping novice programmers solve problems using the PCDIT framework. 2022 55th Hawaii International Conference on System Sciences (HICSS): January 4-7: Proceedings. 982-991. Research Collection School Of Computing and Information Systems. Available at: https://ink.library.smu.edu.sg/sis_research/6845 (best to print hardcopy and read);
  • and the taking of Security Tools Lab 2, cybersecurity module offered by Singapore University of Technology and Design (SUTD) run by Yeaz Elias Jaddoo.

<I also studied Python programming under Cyrille Jegourel and Oka Kurniawan for the ModularMaster in Data Science at SUTD. On a similar note, I’m indebted to my then Java programming instructor, National University of Singapore (NUS), along with those who helped during the course, including my secondary school senior. Indirectly, far too many others made the writing of this even possible.>

*

I am posting here because the PCDIT (Problem Definition, Cases, Design of Algorithm, Implementation, and Testing) framework benefited me even as an adult learner; one with a History degree background, and so I want to share how; and provide a response to its disadvantages.

PCDIT gave me a way to flesh out problems and processes, and sparked by this I intend to look at other forms like pseudocode to develop my critical thinking further.

Moreover, Testing left its great mark as I pushed the boundaries in one memorable Python project to prevent the programme crashing. Over time I’ve come to value such Testing since it helps preclude and pre-empt programme weaknesses that can be exploited by hackers (external or internal).

*

Steps before syntax: Helping novice programmers solve problems using the PCDIT framework referred to the expansive amounts of time required for the process (“one of its disadvantages… consume more time”, p.9 of 11).

In response, one could tackle motivational issues of users/students and reduce their resistance.

  • Highlight the problem solving value of PCDIT
    • cite Madison Kanna: “I had spent two months studying for my first technical interview. I thought I was prepared, but as the interview came to a close, it hit me: I had no idea how to solve coding problems. Of all the tutorials I had taken when I was learning to code, not one of them had included an approach to solving coding problems.” How to Solve Coding Problems with a Simple Four Step Method. 4 Feb 2021. https://www.freecodecamp.org/news/how-to-solve-coding-problems/)
  • Pinpoint potential security exploits from improper testing i.e. extreme scenarios or exceptions / tied to pentration testing

Other Resources

  • [Consider likewise the pseudocode here for learning/reference] Mark Stamp (2018). Introduction to machine learning with applications in information security. CRC Press, Taylor & Francis Group.
  • Kyungbin Kwon. 2017. Novice programmer’s misconception of programming reflected on problem-solving plans. Indiana University. https://files.eric.ed.gov/fulltext/EJ1207584.pdf. US. International Journal of Computer Science Education in Schools, Oct 2017, Vol. 1, No. 4. ISSN 2513-8359. [Education Resources Information Center (ERIC) -> internet-based digital library of education research and information sponsored by the Institute of Education Sciences (IES) of the U.S. Department of Education.]
  • [National Library Board/NLB, Singapore] Ali Almossawi. 2017. Bad choices : how algorithms can help you think smarter and live happier. New York, US : Viking.
  • [NLB] Bradford Tuckfield. 2020. Dive into algorithms : a Pythonic adventure for the intrepid beginner. San Francisco, US : No Starch Press.
  • [NLB; uses C++] V. Anton Spraul. 2012. Think like a programmer : an introduction to creative problem solving. No Starch Press.
  • [NLB; C / C++ / Java / Python] George T. Heineman, Gary Pollice & Stanley Selkow. 2016. Algorithms in a nutshell. Sebastopol, CA. O’Reilly.
  • Oka Kurniawan. What is Computational Thinking? https://www.thelab.sg/what-is-computational-thinking/. The Lab. Singapore.
  • [NLB] Cory Althoff. [2017 or 2022]. The self-taught programmer.
  • [Useful video here on cybersecurity] Towson University. Cyber4all@Towson. https://cisserv1.towson.edu/~cyber4all/.
  • [I’ve not attempted / tested the exercises here but they offer Buffer Overflow courses on C++, Java, and Python. They offer also other cybersecurity courses it seems] Cybersecurity Modules: Security Injections|Cyber4All @Towson. https://cisserv1.towson.edu/~cssecinj/.
  • Safer Computer Coding -> Static Analysis. 24 May 2022. https://chenweilun2014.wordpress.com/2022/05/24/safer-computer-coding-static-analysis/.
  • [NLB] Robert C. Seacord. 2020. Effective c : An introduction to professional c programming. New York, US : No Starch Press.
  • [NLB] Mark G. Graff and Kenneth R. van Wyk. 2003. Secure coding : principles and practices. Sebastopol, California, US: O’Reilly.
  • [NLB, electronic only] Vardan Grigoryan. 2020. Expert c++. Birmingham B3 2PB, UK: Packt Publishing.
  • [NLB, electronic only] Loren Kohnfelder. 2021. Designing secure software : A guide for developers. New York, US : No Starch Press.

Safer Coding -> Static Analysis

/ chenweilun2014 / 1 Comment

# for software developers/engineers; coders; DevOps; risk managers

Static Computer Coding Analysers

“1. examines source code to; 2. detect and report weaknesses that can lead to security vulnerabilities.” [National Institute of Standards and Technology (NIST). updated 15 Feb 2022. Source Code Security Analyzers. https://www.nist.gov/itl/ssd/software-quality-group/source-code-security-analyzers. U.S. Department of Commerce.]

They are also termed Static Application Security Testing (SAST) Tools. Basically they check the programme without running it. Its opposite but complementary number hails as the Dynamic Application Security Testing (DAST). DAST can be done in a sandbox environment, potentially for performance purposes, and not just security ones.

Lists -> analysis tools: various programming languages

Specific Details / Examples

  • [at 36 minute mark] Terry Chia. Static code analysis with Semgrep. [SecWed] 1 Sept 21 | Cyber Risk Quantification + Static code analysis with Semgrep. https://www.youtube.com/watch?v=_OthjwiiebQ&t=2182s. NUS (National University of Singapore) Greyhats. Singapore.
  • Reijo Tomperi. (2009 – 2013). cppcheck – Tool for static C/C++ code analysis. https://linux.die.net/man/1/cppcheck.
  • CERN Computer Security Team. no date. Good Programming in C/C++. https://security.web.cern.ch/recommendations/en/program_c.shtml. Conseil Européen pour la Recherche Nucléaire (CERN), or European Council for Nuclear Research. Esplanade des Particules 1. P.O. Box 1211 Geneva 23. Switzerland.
  • CERN Computer Security Team. no date. Common vulnerabilities guide for C programmers. https://security.web.cern.ch/recommendations/en/codetools/c.shtml. CERN.
  • Rahma Mahmood and Qusay H. Mahmoud. (2018?) Evaluation of Static Analysis Tools for Finding Vulnerabilities in Java and C/C++ Source Code. https://arxiv.org/ftp/arxiv/papers/1805/1805.09040.pdf. Department of Electrical, Computer & Software Engineering. University of Ontario Institute of Technology. Oshawa, ON, Canada. [arXiv is a free distribution service and an open-access archive for 2,070,036 scholarly articles in the fields of physics, mathematics, computer science, quantitative biology, quantitative finance, statistics, electrical engineering and systems science, and economics. Materials on this site are not peer-reviewed by arXiv… arXiv was founded by Paul Ginsparg in 1991 and is now maintained and operated by Cornell Tech].
  • [Bandit SAST for Python] Encrypting PDF files – Python. https://chenweilun2014.wordpress.com/2022/04/27/encrypting-pdf-files/.

The hundred-page machine learning book / 2019

/ chenweilun2014 / 1 Comment

Andriy Burkov wrote this book and self-published it (Quebec City, Canada).

Why read his book? It helps you understand concepts and gain insights even if you have gone through Data Science/Machine Learning courses. Consider what Gareth James (one of the authors for An Introduction to Statistical Learning, with Applications in R) penned in the Foreword, plus what Peter Norvig former Director of Research at Google recorded on the back cover.

Now to the notes…

  • What differentiates shallow from deep learning?
  • [Chapter 3] Five most important algorithms
    • linear regression [cause-effect / prediction]
    • logistic regression [correctly and beautifully explained as ‘classifier’]
    • decision tree
    • support vector machine (SVM)
    • k-nearest neighbors (KNN)
  • [Chapter 5] z-score normalisation / standardisation for features (input variable to generate predictions – https://developers.google.com/machine-learning/glossary#feature)
    • useful for unsupervised learning
    • features tending towards normal distribution
    • features with large scale outliers (not great volume)
    • else use normalisation
    • if you have time and bandwidth, Burkov suggests trying both out on your data to see which performs better
  • [Chapter 5] Choosing algorithms (selected criteria only)
  • [Chapter 5] Underfitting -> poor predictive power (training data)
    • model too simplistic (linear model prone to underfit)
    • features do not predict output/dependent variable/label
    • solve: complex model and re-engineer features
  • [Chapter 5] Overfitting -> poor prediction in test or validation sets i.e. high variance
    • excessively composite or complicated model (e.g. too tall a decision tree; neural network too wide/deep)
    • vast number of features + too few training examples
    • solve: simplify model (e.g. polynomial -> linear models; decrease units/layers in neural network); decrease dimensionality of examples (https://developers.google.com/machine-learning/glossary?hl=en#example); enlarge training set; regularise model
  • [Chapter 7] ensemble learning algorithms include gradient boosting (https://www.geeksforgeeks.org/ml-gradient-boosting/) and random forest (bagging family: decreased variance leads to decreased overfitting); may offer more precision than those from Chapter 3
  • [Chapter 7] Sequence-to-Sequence Learning:
    • for translation; room for more development
    • most effectively tackled by neural networks
    • process of training encoder and decoder at the same time (training data)
  • [Chapter 8] Imbalanced datasets: you can use random forest (ensemble algorithm), gradient boosting, decision tree
  • [Chapter 8] Improve model performance via combination of two or two unlinked models e.g. SVM and random forest.
  • [Chapter 9] unsupervised learning
    • clustering; dimensionality reduction; outlier detection (one-class classifier: one-class Gaussian/ k-means/ KNN/ SVM or autoencoder)
  • [Chapter 9] HDBSCAN (Hierarchical Density-Based Spatial Clustering of Applications with Noise)> DBSCAN (Density-Based Spatial Clustering and Application with Noise)
  • [Chapter 10] Word Embeddings / Self-supervised learning
    • skip-gram (word2vec algorithm)
    • generate model translates one-hot encoding of words into word embedding

Related

Microsoft PowerShell -> Pros/Cons

/ chenweilun2014 / Leave a comment

Been wanting to write this since late 2020!

Beginning a Data Science course, I learnt PowerShell to verify the hash (identification number basically) of Anaconda, a data science platform that hosts tools like Spyder and Jupyter Notebook, for authentication. If the hash failed to match, it could have been a fake or malicious file meant to attack my device.

It was exciting! (with guidance from — Sean O’Shea. 2 Jul 2019. PowerShell Command to get Hash Values. https://www.litigationsupporttipofthenight.com/single-post/2019/07/01/powershell-command-to-get-hash-values; SentinelOne. What is Hashing? https://www.sentinelone.com/cybersecurity-101/hashing/).

But PowerShell can be used for so many other things! Note though there are different versions (see https://www.techrepublic.com/article/a-tale-of-two-powershells-which-is-the-right-version-for-you/; https://docs.microsoft.com/en-us/powershell/scripting/whats-new/differences-from-windows-powershell?view=powershell-7.1)

With great power comes great responsibility right? PowerShell has become an attack channel for internal and external threats. Hence, disabling it for non-administrative users would be wise. Consider the below for your reference and action:

? Code – Password Cracker

/ chenweilun2014 / Leave a comment

Read Ethical Hacking with Python (https://www.geeksforgeeks.org/ethical-hacking-with-python/), 12 Dec 2019. This provided code for password cracking.

It reminded me of tools I used earlier in a cybersecurity project.

Running these as an ethical hacker can pinpoint weakness (vulnerabilities) for mitigation.

Associated