<?xml version="1.0" encoding="utf-8"?><feed xmlns="http://www.w3.org/2005/Atom" ><generator uri="https://jekyllrb.com/" version="3.10.0">Jekyll</generator><link href="https://mike.cloud/feed.xml" rel="self" type="application/atom+xml" /><link href="https://mike.cloud/" rel="alternate" type="text/html" /><updated>2024-12-06T08:25:29+00:00</updated><id>https://mike.cloud/feed.xml</id><title type="html">Mike Klimek</title><subtitle>Welcome to my blog about development and tech.</subtitle><author><name>Mike Klimek</name></author><entry><title type="html">Large Language Models</title><link href="https://mike.cloud/software%20design/2024/10/12/custom-llms.html" rel="alternate" type="text/html" title="Large Language Models" /><published>2024-10-12T00:00:00+00:00</published><updated>2024-10-12T00:00:00+00:00</updated><id>https://mike.cloud/software%20design/2024/10/12/custom-llms</id><content type="html" xml:base="https://mike.cloud/software%20design/2024/10/12/custom-llms.html"><![CDATA[<p>This page is intended as a simplified primer on important concepts for developing and customizing Large Language Models (LLM).</p>

<details>
  <summary>Models</summary>

  <p>Training a completely new <strong>base model</strong> is <strong>extremely expensive</strong> (GPT-4: $100M) due to the required compute power to achieve good results, so this is only done by tech companies with enough capital:</p>

  <ul>
    <li>Meta: <a href="https://huggingface.co/meta-llama">Llama</a> (public)</li>
    <li>Google: <a href="https://huggingface.co/collections/google/gemma-2-release-667d6600fd5220e7b967f315">Gemma</a> (public), <a href="https://ai.google.dev/gemini-api">Gemini</a> (API only)</li>
    <li>OpenAI: <a href="https://platform.openai.com/docs/models/overview">GPT</a> (API only)</li>
    <li>Anthropic: <a href="https://www.anthropic.com/claude">Claude</a> (API only)</li>
    <li>Mistral AI: <a href="https://huggingface.co/docs/transformers/main/en/model_doc/mistral">Mistral</a> (public)</li>
    <li>(Many other models and remixes of larger models)</li>
  </ul>

  <p>These models are trained on <a href="https://commoncrawl.org/">billions of web pages</a>, e.g. wikis, open source projects, etc.</p>

  <p>These models often have different <strong>variants</strong>:</p>
  <ul>
    <li>Different sizes (e.g. 2B, 70B or 405B parameters). Larger models are <a href="https://lmarena.ai/?leaderboard">more powerful</a>, but require more resources.</li>
    <li>Fine tunings (e.g. “instruct models” optimized for chatbots)</li>
    <li>Multimodal models (understand images or speech)</li>
  </ul>

  <p>Models have different <strong>hardware requirements</strong>:</p>
  <ul>
    <li>Small models (≤ 70B): Can run <strong>locally</strong> (no internet access) on consumer hardware (gaming graphics cards, M1 MacBooks).</li>
    <li>Large models: Require expensive specialized hardware so <strong>cloud access</strong> is usually cheaper.</li>
  </ul>
</details>

<details>
  <summary>Input/Output</summary>

  <p>LLMs are simple when viewed from the outside:</p>
  <ul>
    <li><strong>Input</strong>: Text</li>
    <li><strong>Output</strong>: Text</li>
  </ul>

  <p>E.g. when the input is an entire chat history in a chat-bot, the LLM can predict what the next word(s) of the text would look like (more chat messages).</p>

  <p>By cleverly choosing the input (<strong>Prompt Engineering</strong>), you can influence the output, e.g. with examples, instructions for language patterns, more context, etc.</p>
</details>

<details>
  <summary>System Prompts</summary>

  <p>System prompts are predefined instructions given to an LLM in addition to user’s input. They set the context and tone for the interaction.</p>

  <p>Examples:</p>
  <ul>
    <li><strong>Role</strong>: “You are a helpful assistant.”</li>
    <li><strong>Behavior</strong>: “Answer in English and provide detailed explanations.”</li>
    <li><strong>Context</strong>: “The time is {time} and the user’s name is {name}. Here is a document relevant to the user’s query: {document}”</li>
  </ul>
</details>

<details>
  <summary>Vector Databases</summary>

  <p>Data can be stored in vector databases via mathematical “feature vectors”. A collection of vectors is called an <strong>embedding</strong>.</p>

  <p>These vectors lie in a multidimensional <strong>vector space</strong> (often plotted as a point cloud). Each dimension describes a <strong>semantic meaning</strong> (feature) of a data element. This has the advantage that related pieces of data are close to each other in the vector space.</p>

  <p><strong>Semantic search</strong>: A search query is converted to a feature vector and neighbors of this vector (related data) are retrieved.</p>

  <p><strong>Embedding Models</strong>: Text is converted to feature vectors with specialized embedding models, e.g. <a href="https://ollama.com/library/nomic-embed-text">nomic-embed-text</a>.</p>

  <p><img src="https://ds055uzetaobb.cloudfront.net/brioche/uploads/JERsKXkW4T-screen-shot-2016-05-05-at-123118-pm.png?width=2400" alt="" />
<em>Source: Gutierrez-Osuna, R. Introduction to Pattern Analysis</em></p>

  <p><img src="https://developers.google.com/static/machine-learning/crash-course/embeddings/images/linear_relationships.svg" alt="" />
<em>Source: <a href="https://developers.google.com/machine-learning/crash-course/embeddings/embedding-space">Google Developer - Machine Learning Crash Course</a></em></p>
</details>

<details>
  <summary>Retrieval Augment Generation</summary>

  <p>Retrival augmented generation (RAG) is used to enrich a model with knowledge from documents (e.g. PDFs, videos, websites, …).</p>

  <p>These documents are split into chunks that are stored in a <strong>vector database</strong> as <strong>embeddings</strong>. Embeddings need to be regenerated if the documents change, so embeddings are <strong>not real-time capable</strong>.</p>

  <p>A user request proceeds as follows:</p>
  <ul>
    <li>The request is converted into a vector in the embedding’s vector space.</li>
    <li>Neighbors of this vector are searched in the vector database (<strong>semantic search</strong>).</li>
    <li>These vectors are <strong>added to the system prompt</strong> in their original text form.</li>
    <li>The LLM generates a response based on the additional context in the prompt.</li>
  </ul>
</details>

<details>
  <summary>Tool Calling</summary>

  <p>Tool calling allows an LLM to call predefined <strong>functions</strong> (“tools”) in the executing framework’s code. With this it can call external APIs (search engines, databases, triggers, etc.). Therefore they are <strong>real-time capable</strong>.</p>

  <p>LLMs still only handle text input and output:</p>
  <ul>
    <li>A <strong>text description</strong> for each tool is added to the LLMs prompt, so it knows how to use them correctly.</li>
    <li>It can emit <strong>JSON</strong> formatted text during generation.</li>
    <li>The framework intercepts the output and calls the tool.</li>
    <li>The tools result is then <strong>added to the system prompt</strong> and the LLM can continue with the next tokens.</li>
  </ul>

  <p>This works better if the model was fine tuned for tool calling.</p>

  <p>Example:</p>

  <div class="language-js highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">const</span> <span class="nx">getRecipesByMainIngedrientTool</span><span class="p">:</span> <span class="nx">RunnableToolFunction</span><span class="o">&lt;</span><span class="nx">any</span><span class="o">&gt;</span> <span class="o">=</span> <span class="p">{</span>
    <span class="na">type</span><span class="p">:</span> <span class="dl">'</span><span class="s1">function</span><span class="dl">'</span><span class="p">,</span>
    <span class="na">function</span><span class="p">:</span> <span class="p">{</span>
        <span class="na">description</span><span class="p">:</span> <span class="dl">'</span><span class="s1">Search recipes by main ingredient. Ingredients should be in English and snakecase.</span><span class="dl">'</span><span class="p">,</span>
        <span class="na">function</span><span class="p">:</span> <span class="nx">recipeApi</span><span class="p">.</span><span class="nx">getRecipesByMainIngredient</span><span class="p">,</span>
        <span class="na">parameters</span><span class="p">:</span> <span class="p">{</span>
            <span class="na">type</span><span class="p">:</span> <span class="dl">'</span><span class="s1">object</span><span class="dl">'</span><span class="p">,</span>
            <span class="na">properties</span><span class="p">:</span> <span class="p">{</span> <span class="na">ingredient</span><span class="p">:</span> <span class="p">{</span> <span class="na">type</span><span class="p">:</span> <span class="dl">'</span><span class="s1">string</span><span class="dl">'</span> <span class="p">}</span> <span class="p">},</span>
        <span class="p">},</span>
        <span class="na">parse</span><span class="p">:</span> <span class="nx">JSON</span><span class="p">.</span><span class="nx">parse</span><span class="p">,</span>
    <span class="p">},</span>
<span class="p">};</span>
</code></pre></div>  </div>

</details>

<details>
  <summary>Fine Tuning</summary>

  <p>Fine-tuning adjusts the <strong>general behavior</strong> of an existing model through <strong>further training</strong>. For example to use it as a chatbot (question/answer), as a code generator, to write books, or to use certain vocabulary.</p>

  <p>There are two options for this:</p>
  <ul>
    <li><strong>Full parameter fine-tuning</strong>: An existing model is further trained, but with a specialized dataset. This adjusts <strong>all parameters</strong> over time, creating a new model. Requires a lot of computing power (expensive).</li>
    <li><strong>Low-rank adaptation (LoRA)</strong>: Adds <strong>additional parameters</strong> to a model without affecting the original parameters. Requires less computing power.</li>
  </ul>
</details>]]></content><author><name>Mike Klimek</name></author><category term="Software Design" /><summary type="html"><![CDATA[An primer on important concepts you should know when building and customizing AI assistants.]]></summary></entry><entry><title type="html">Tips &amp;amp; Tricks</title><link href="https://mike.cloud/android/2022/11/06/tips-and-tricks.html" rel="alternate" type="text/html" title="Tips &amp;amp; Tricks" /><published>2022-11-06T00:00:00+00:00</published><updated>2022-11-06T00:00:00+00:00</updated><id>https://mike.cloud/android/2022/11/06/tips-and-tricks</id><content type="html" xml:base="https://mike.cloud/android/2022/11/06/tips-and-tricks.html"><![CDATA[<p>Some short and sweet tips, tricks and best practices for Android development:</p>

<details>
  <summary>Using the Command Line</summary>

  <p>Some examples:</p>
  <div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># Build release APK</span>
./gradlew assembleRelease
<span class="c"># Build release AAB</span>
./gradlew bundleRelease
<span class="c"># Run tests</span>
./gradlew testReleaseUnitTest
<span class="c"># Install APK on device</span>
abd <span class="nb">install </span>my_app.apk
<span class="c"># List connected devices</span>
adb devices
<span class="c"># Connect over wifi (enabled in developer menu)</span>
adb pair &lt;ip&gt;:&lt;port&gt;
<span class="c"># Open shell on device</span>
adb shell
<span class="c"># Enter text</span>
adb shell input text foo
<span class="c"># Press "enter" key</span>
adb shell input keyevent 66
<span class="c"># Simulate process death (app must be in background)</span>
adb shell am <span class="nb">kill </span>my.application.id
<span class="c"># Take screenshot</span>
adb exec-out screencap <span class="nt">-p</span> <span class="o">&gt;</span> ./screen.png
<span class="c"># Test deep link</span>
adb shell am start <span class="nt">-a</span> android.intent.action.VIEW <span class="nt">-d</span> https://my.url.io/my_file
<span class="c"># Show log cat output</span>
adb logcat <span class="nt">-v</span> color,brief <span class="nt">--pid</span><span class="o">=</span><span class="si">$(</span>adb shell pidof my.application.id<span class="si">)</span>
<span class="c"># Open a Kotlin REPL</span>
kotlin
<span class="c"># Run a Kotlin script</span>
kotlinc <span class="nt">-script</span> my_script.kts
</code></pre></div>  </div>

  <p>Development on command line requires a JDK (Java Development Kit) and Android SDK in your <code class="language-plaintext highlighter-rouge">$PATH</code>.</p>

  <p>Android Studio contains an embedded JDK (and a Kotlin compiler), no need to install it separately. Using the embedded JDK also has some other advantages (e.g. command line and Android Studio use the same Gradle daemon).</p>

  <p>On MacOS just add this to your <code class="language-plaintext highlighter-rouge">~/.zshrc</code>:</p>

  <div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># Android</span>
<span class="nb">export </span><span class="nv">ANDROID_HOME</span><span class="o">=</span><span class="s2">"/Users/&lt;YOUR USER NAME HERE&gt;/Library/Android/sdk"</span>
<span class="nb">export </span><span class="nv">PATH</span><span class="o">=</span><span class="s2">"</span><span class="k">${</span><span class="nv">PATH</span><span class="k">}</span><span class="s2">:</span><span class="k">${</span><span class="nv">ANDROID_HOME</span><span class="k">}</span><span class="s2">/tools"</span>
<span class="nb">export </span><span class="nv">PATH</span><span class="o">=</span><span class="s2">"</span><span class="k">${</span><span class="nv">PATH</span><span class="k">}</span><span class="s2">:</span><span class="k">${</span><span class="nv">ANDROID_HOME</span><span class="k">}</span><span class="s2">/tools/bin"</span>
<span class="nb">export </span><span class="nv">PATH</span><span class="o">=</span><span class="s2">"</span><span class="k">${</span><span class="nv">PATH</span><span class="k">}</span><span class="s2">:</span><span class="k">${</span><span class="nv">ANDROID_HOME</span><span class="k">}</span><span class="s2">/platform-tools"</span>

<span class="c"># Kotlin</span>
<span class="c"># Note: You may have to give the binaries execution permission: chmod +x kotlinc</span>
<span class="nb">export </span><span class="nv">KOTLIN_HOME</span><span class="o">=</span><span class="s2">"/Applications/Android Studio.app/Contents/plugins/Kotlin/kotlinc"</span>
<span class="nb">export </span><span class="nv">PATH</span><span class="o">=</span><span class="s2">"</span><span class="k">${</span><span class="nv">PATH</span><span class="k">}</span><span class="s2">:</span><span class="k">${</span><span class="nv">KOTLIN_HOME</span><span class="k">}</span><span class="s2">/bin"</span>

<span class="c"># Java (no need to export to $PATH)</span>
<span class="nb">export </span><span class="nv">JAVA_HOME</span><span class="o">=</span><span class="s2">"/Applications/Android Studio.app/Contents/jre/Contents/Home"</span>
</code></pre></div>  </div>
</details>

<details>
  <summary>Screen Mirroring</summary>

  <p>I highly recommend <a href="https://github.com/Genymobile/scrcpy">scrcpy</a>, if you prefer working with a real device or can’t use an emulator (e.g. for Bluetooth apps).</p>

  <p>Features:</p>
  <ul>
    <li>Mirror your device’s screen on your desktop: <code class="language-plaintext highlighter-rouge">scrcpy</code></li>
    <li>Take recordings: <code class="language-plaintext highlighter-rouge">scrcpy --record recording.mp4</code></li>
    <li>You can use your keyboard to type on your device or click anywhere with your mouse.</li>
    <li>Shared clipboard, so you can copy and past from and to you device.</li>
    <li>Drag and drop to transfer files or install APKs.</li>
  </ul>
</details>

<details>
  <summary>Signing Keys</summary>

  <p>You can use <code class="language-plaintext highlighter-rouge">keytool</code> for creating a keystore and adding signing keys to it. It is part of the JDK. This can also also be done in Android Studio (Build → Generate Signed APK).</p>

  <div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># Create keystore or add key to existing keystore</span>
keytool <span class="nt">-v</span> <span class="nt">-genkey</span> <span class="nt">-keystore</span> ./keystore.jks <span class="nt">-keyalg</span> RSA <span class="nt">-keysize</span> 2048 <span class="nt">-validity</span> <span class="k">$((</span><span class="m">365</span> <span class="o">*</span> <span class="m">50</span><span class="k">))</span> <span class="nt">-alias</span> MyKey <span class="nt">-deststoretype</span> jks
<span class="c"># List keys in keystore</span>
keytool <span class="nt">-v</span> <span class="nt">-list</span> <span class="nt">-keystore</span> ./keystore.jks
<span class="c"># Move key from one keystore to another (creates other if it does not exist)</span>
keytool <span class="nt">-v</span> <span class="nt">-importkeystore</span> <span class="nt">-srckeystore</span> keystore.jks <span class="nt">-destkeystore</span> new_keystore.jks <span class="nt">-srcalias</span> MyKey <span class="nt">-destalias</span> MyKey <span class="nt">-deststoretype</span> jks
</code></pre></div>  </div>
</details>

<details>
  <summary>Drawable Resources</summary>

  <p>Use the <code class="language-plaintext highlighter-rouge">drawable-nodpi</code> resource directory if you only have one size of an image. The <code class="language-plaintext highlighter-rouge">drawable</code> resource directory is the same as <code class="language-plaintext highlighter-rouge">drawable-mdpi</code> and will scale images up on devices with a higher pixel density. A 1600 x 1000 image <a href="https://medium.com/@oronno/android-drawable-outofmemoryerror-ebe2995760b6">will be scaled up</a> to 6400 x 4000 on an xxxhdpi device. This can quickly lead to <code class="language-plaintext highlighter-rouge">OutOfMemoryError</code>.</p>

  <p>For vector assets this is not a problem. However you can use the <code class="language-plaintext highlighter-rouge">drawable-anydpi</code> resource directory, which will only be used if no resource is defined for any other density.</p>
</details>

<details>
  <summary>Splash Screen</summary>

  <p>You can add a simple splash-screen to you app, by adding a splash theme, which will be shown until the launched activity is fully loaded:</p>

  <div class="language-xml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c">&lt;!-- styles.xml --&gt;</span>
<span class="nt">&lt;style</span> <span class="na">name=</span><span class="s">"MyAppTheme.Splash"</span><span class="nt">&gt;</span>
    <span class="nt">&lt;item</span> <span class="na">name=</span><span class="s">"android:windowBackground"</span><span class="nt">&gt;</span>@drawable/splash<span class="nt">&lt;/item&gt;</span>
<span class="nt">&lt;/style&gt;</span>
</code></pre></div>  </div>

  <div class="language-xml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c">&lt;!-- splash.xml --&gt;</span>
<span class="nt">&lt;layer-list</span> <span class="na">xmlns:android=</span><span class="s">"http://schemas.android.com/apk/res/android"</span> <span class="na">android:opacity=</span><span class="s">"opaque"</span><span class="nt">&gt;</span>
    <span class="nt">&lt;item</span> <span class="na">android:drawable=</span><span class="s">"?android:colorBackground"</span> <span class="nt">/&gt;</span>
    <span class="nt">&lt;item</span> <span class="na">android:drawable=</span><span class="s">"@drawable/ic_launcher_foreground"</span> <span class="na">android:gravity=</span><span class="s">"center"</span> <span class="nt">/&gt;</span>
<span class="nt">&lt;/layer-list&gt;</span>
</code></pre></div>  </div>

  <div class="language-xml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c">&lt;!-- AndroidManifest.xml --&gt;</span>
<span class="nt">&lt;activity</span>
    <span class="na">android:name=</span><span class="s">".MainActivity"</span>
    <span class="na">android:theme=</span><span class="s">"@style/MyAppTheme.Splash"</span> <span class="nt">/&gt;</span>
</code></pre></div>  </div>

  <div class="language-kotlin highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1">// MainActivity.kt</span>
<span class="k">override</span> <span class="k">fun</span> <span class="nf">onCreate</span><span class="p">(</span><span class="n">savedInstanceState</span><span class="p">:</span> <span class="nc">Bundle</span><span class="p">?)</span> <span class="p">{</span>
    <span class="nf">setTheme</span><span class="p">(</span><span class="nc">R</span><span class="p">.</span><span class="n">style</span><span class="p">.</span><span class="nc">MyAppTheme</span><span class="p">)</span> <span class="c1">// app is started, so we can now remove the splash screen theme</span>
    <span class="k">super</span><span class="p">.</span><span class="nf">onCreate</span><span class="p">(</span><span class="n">savedInstanceState</span><span class="p">)</span>
    <span class="nf">setContentView</span><span class="p">(</span><span class="nc">R</span><span class="p">.</span><span class="n">layout</span><span class="p">.</span><span class="n">activity_main</span><span class="p">)</span>
<span class="p">}</span>
</code></pre></div>  </div>
</details>

<details>
  <summary>Formatting (Dates, Currency, etc.)</summary>

  <p>You should use the user’s locale specific date (<code class="language-plaintext highlighter-rouge">12/31/22</code> vs. <code class="language-plaintext highlighter-rouge">31.12.22</code>), time, currency and number (<code class="language-plaintext highlighter-rouge">10,000.00</code> vs <code class="language-plaintext highlighter-rouge">10.000,00</code>, <a href="https://en.wikipedia.org/wiki/Eastern_Arabic_numerals">Eastern Arabic numerals</a>) formats.</p>

  <ul>
    <li>Dates:
      <ul>
        <li>Short: <code class="language-plaintext highlighter-rouge">DateFormat.getDateFormat(context).format(date)</code></li>
        <li>Medium: <code class="language-plaintext highlighter-rouge">DateFormat.getMediumDateFormat(context).format(date)</code></li>
        <li>Long: <code class="language-plaintext highlighter-rouge">DateFormat.getLongDateFormat(context).format(date)</code></li>
      </ul>
    </li>
    <li>Time: <code class="language-plaintext highlighter-rouge">DateFormat.getTimeFormat(context).format(date)</code></li>
    <li>Percentage: <code class="language-plaintext highlighter-rouge">NumberFormat.getPercentInstance().format(percentage)</code></li>
    <li>Numbers: <code class="language-plaintext highlighter-rouge">NumberFormat.getNumberInstance().format(number)</code></li>
    <li>Currency:
      <ul>
        <li>German (Euro): <code class="language-plaintext highlighter-rouge">NumberFormat.getCurrencyInstance(Locale.GERMANY).format(amount)</code></li>
        <li>US (Dollar): <code class="language-plaintext highlighter-rouge">NumberFormat.getCurrencyInstance(Locale.US).format(amount)</code></li>
        <li>User locale (Euro): <code class="language-plaintext highlighter-rouge">NumberFormat.getCurrencyInstance().apply { setCurrency(Currency.getInstance("EUR")) }.format(amount)</code></li>
      </ul>
    </li>
  </ul>
</details>

<details>
  <summary>System Windows</summary>

  <p>You can draw the app’s background behind system windows (status bar, camera notches, keyboard) by setting <code class="language-plaintext highlighter-rouge">android:windowTranslucentStatus="true"</code> on your app’s theme.</p>

  <p>To prevent elements like text or buttons from also drawing behind the status bar, they need to receive some padding. This padding is called “window insets”. It usually has different values for all 4 sides is provided by the system at runtime, e.g. when the keyboard is expanded or when device has a notch (at the top in portrait or at the side in landscape mode).</p>

  <p>To automatically apply this padding you can set <code class="language-plaintext highlighter-rouge">android:fitsSystemWindows="true"</code> on a view or use <code class="language-plaintext highlighter-rouge">Modifier.windowInsetsPadding(WindowInsets.systemBars)</code> in Jetpack Compose. The insets will be consumed by the view and no other will receive it, so it is best applied to a view group, like your root layout. These insets will override any other paddings you have defined on the view.</p>
</details>

<details>
  <summary>Right-to-Left Locales</summary>

  <p>Used for Arabic and Hebrew.</p>
  <ul>
    <li>Make sure you have <code class="language-plaintext highlighter-rouge">android:supportsRtl="true"</code> in your Manifest.</li>
    <li>Always use “start” and “end” instead of “left” and “right”, e.g <code class="language-plaintext highlighter-rouge">layout_marginStart</code>, <code class="language-plaintext highlighter-rouge">layout_constraintStart_toEndOf</code>, <code class="language-plaintext highlighter-rouge">layout_gravity="end"</code>. If you do this consistently, your layouts should look good on RTL locales without much extra work.</li>
    <li>You can automatically mirror vector assets (e.g. left/right arrows) for RTL locales by using <code class="language-plaintext highlighter-rouge">android:autoMirrored="true"</code>. No logic or separate drawable in <code class="language-plaintext highlighter-rouge">drawable-ldrtl</code> needed.</li>
    <li>Use <a href="https://developer.android.com/jetpack/androidx/releases/viewpager2">ViewPager2</a>, which automatically switches scroll direction for RTL locales.</li>
    <li>You can quickly check your XML layouts in Android Studio by selecting “Preview Right to Left” under the locale selection in the layout preview.</li>
  </ul>
</details>

<details>
  <summary>XML Layouts</summary>

  <p>These tips have become obsolete with Jetpack Compose, but may be helpful for existing apps:</p>
  <ul>
    <li>For some easy animations add <code class="language-plaintext highlighter-rouge">android:animateLayoutChanges="true"</code> on parent layouts, so changes to their children (e.g. visibility) are animated.</li>
    <li>You can add fading edges on scrollable views, so content disappears smoothly at the top and bottom with <code class="language-plaintext highlighter-rouge">android:requiresFadingEdge="vertical"</code> and <code class="language-plaintext highlighter-rouge">android:fadingEdgeLength="8dp"</code>.</li>
    <li>You can add dividers between <code class="language-plaintext highlighter-rouge">LinearLayout</code> children with <code class="language-plaintext highlighter-rouge">android:showDividers="middle"</code> and <code class="language-plaintext highlighter-rouge">android:divider="?dividerHorizontal"</code>.</li>
    <li>You can set a TextView to <a href="https://developer.android.com/develop/ui/views/text-and-emoji/autosizing-textview">automatically</a> shrink its font size, so the text will shrink to always fit with <code class="language-plaintext highlighter-rouge">android:autoSizeTextType="uniform"</code>. Do not use <code class="language-plaintext highlighter-rouge">wrap_content</code> for width or height or it will not work correctly.</li>
  </ul>
</details>]]></content><author><name>Mike Klimek</name></author><category term="Android" /><summary type="html"><![CDATA[Short and sweet tips, tricks and best practices for Android development.]]></summary></entry><entry><title type="html">Autonomous Teams</title><link href="https://mike.cloud/tech%20industry/2022/06/24/autonomous-teams.html" rel="alternate" type="text/html" title="Autonomous Teams" /><published>2022-06-24T00:00:00+00:00</published><updated>2022-06-24T00:00:00+00:00</updated><id>https://mike.cloud/tech%20industry/2022/06/24/autonomous-teams</id><content type="html" xml:base="https://mike.cloud/tech%20industry/2022/06/24/autonomous-teams.html"><![CDATA[<p>If you keep an eye on German automotive companies, you will see a pattern. Their software <a href="https://www.focus.de/auto/news/pleiten-pech-und-pannen-bei-cariad-software-desaster-fuer-herbert-diess-warum-der-stuhl-des-vw-chefs-wackelt_id_97501221.html">is a huge mess</a> and they have <a href="https://www.spiegel.de/wirtschaft/unternehmen/elektro-offensive-mercedes-sucht-3000-softwareingenieure-a-bc5bcb34-a2c3-4151-978c-31ea89591680">trouble finding</a> and <a href="https://www.handelsblatt.com/unternehmen/autobranche-herbert-diess-geraet-unter-druck-kritik-an-vw-softwareeinheit-cariad-waechst/28264920.html">keeping</a> developers, which leads to a high turnover rate, that amplifies existing issues.</p>

<p>In my opinion their failure is deeply rooted in the way these traditional companies are structured and how they see the role of developers. Actually solving the problem, e.g. through <strong>autonomous product teams</strong>, would require a massive change in company structure and culture.</p>

<div class="message">

  <p><strong>Note:</strong> The post might be a bit of rant due to personal experience 😅</p>
</div>

<h1 id="traditional-companies">Traditional Companies</h1>
<ul>
  <li>Traditional companies have a deep hierarchy with lots of middle management layers and external outsourcing:
    <ul>
      <li>Developers, QA and UX/UI are at the bottom, like classical factory workers.</li>
      <li>Decisions are made at the top by people without technical background and little contact (sometimes only presentations) with the software they are managing.</li>
    </ul>
  </li>
  <li>There is no flexibility or room for alternative solutions (outside of R&amp;D), as everything has to be planned and rubber-stamped in advance (waterfall). There is no way a <a href="https://www.theverge.com/2015/10/8/9481651/volkswagen-congressional-hearing-diesel-scandal-fault">few rogue engineers</a> can just do what they want.</li>
  <li>These companies are suffocating on their own processes. There are many obstacles, privilege restrictions, custom tooling and communication channels involved, so it can take weeks to get even minor things done. Often these companies group out their own technical startups, to get out of this process hell for a short time.</li>
  <li>There is a lot of irrational company politics and unnecessary drama involved. This can be amusing from the sideline, until you find yourself in the line of fire.</li>
  <li><a href="https://en.wikipedia.org/wiki/Conway%27s_law">Conway’s law</a> is in full effect. Teams are isolated silos and so is their code. This leads to a severe lack of information transparency, difficult integrations and friction losses. Usually there is no <a href="https://martinfowler.com/bliki/CodeOwnership.html">collective or weak code ownership</a>, so it is discouraged to contribute to other teams’ code, even if you depend on their work.</li>
</ul>

<h1 id="why-exactly-is-this-so-wrong">Why exactly is this so wrong?</h1>
<p>First I strongly recommend reading this blog post by Gergely Orosz: <a href="https://blog.pragmaticengineer.com/what-silicon-valley-gets-right-on-software-engineers/">What Silicon Valley “Gets” about Software Engineers that Traditional Companies Do Not</a>. It perfectly captures the issues that traditional companies have:</p>
<ul>
  <li><strong>Autonomy for software engineers</strong>: Teams have no say in their product, everything is dictated from the top. It’s hard to care about something you can’t change or improve.</li>
  <li><strong>Curious problem solvers, not mindless resources</strong>: The companies see developers more like classical assembly line workers, so they are looking for a <a href="https://www.spiegel.de/wirtschaft/unternehmen/elektro-offensive-mercedes-sucht-3000-softwareingenieure-a-bc5bcb34-a2c3-4151-978c-31ea89591680">huge quantity</a> of “code monkeys” instead of skilled problem solvers. Their potential is wasted, because <a href="https://event-driven.io/en/bring_me_problems_not_solutions/">instead of business problems to solve, they are given solutions</a> to implement without any context. Experiments or mistakes are usually discouraged.</li>
  <li><strong>Internal data, code, and documentation transparency</strong>: Access is restricted as much as possible by default. If you can’t find the documentation you are looking for, than you can not be sure wether you don’t have access to it, or if it just does not exist. Documentation is sometimes written for the process and not for other developers.</li>
  <li><strong>Exposure to the business and to business metrics</strong>: Business metrics are intended for the higher layers, because they make the decisions. Developers don’t know if the feature they worked on for weeks is actually being used, which can be demotivating. Even if the team has access to metric, they can not shift priority on features that create more value.</li>
  <li><strong>Engineer-to-engineer comms over triangle-communication</strong>: There are innumerable communication channels (i.e. no single open point of access like a group channel for every team) and it is not easy to find out who is responsible. So communication always goes over project managers. Upward communication (jumping the chain of command) is discouraged.</li>
  <li><strong>Investing in a less frustrating developer experience</strong>: Instead of providing developers with well maintained infrastructure, the company dictates permissible technology, workflows, and internal tooling (sometimes poorly maintained). Fixing tooling issues can take a considerable amount of time, so bad workarounds are common (ignoring certificates, building manually instead of CI/CD, not using open source libs instead of dealing with legal stoppers), so development does not completely grind to a halt. Managment does not know or care about the impact, because it does not appear in their metrics, or in some cases it is just waved aside because “developers like to complain a lot”.</li>
  <li><strong>Higher leverage –&gt; higher {autonomy, pay}</strong>: Pay grade is usually bound to the level in the hierarchy, so developers have lower pay than managers. In contrast developers at silicon valley receive <a href="https://www.levels.fyi/company/Google/salaries/">higher pay</a> than non technical project managers (though there is more of a focus on strong tech leads, who have even higher pay).</li>
</ul>

<h1 id="how-to-do-it-right">How to do it right</h1>
<p>In my opinion these traditional companies really need small <strong>autonomous product teams</strong> with a <strong>shared vision</strong> and strong <strong>technical leads</strong>. Or with more buzzwords: Self-organized cross-functional agile teams and flat hierarchies.</p>

<p>This actually uses the teams’ full potential:</p>
<ul>
  <li>Product teams know the product they are developing inside out, can estimate costs (e.g. complexity) and see benefits (via business metrics). With this they can find better suited creative solutions to business problems, that minimize cost and maximize benefit.</li>
  <li>They have the necessary flexibility to quickly react to changing circumstances and requirements.</li>
  <li>They are able to optimize their own development flows and collaboration with other teams, which raises motivation and productivity.</li>
</ul>

<h1 id="success-stories">Success Stories</h1>
<p><a href="https://chethaase.medium.com/androids-765c803d5ff6">Androids</a> by Chet Haase describes the success of the team that build the Android OS. One thing that stands out is, that they were completely autonomous within Google (though they still had access to Google’s resources). They made sure to stay free from the rest of Google (processes and company politics) and always followed their long term vision. They were even decoupled from Google’s hiring processes, which would have filtered out a lot of skilled OS developers.</p>

<p>Developer lead teams are <a href="https://blog.pragmaticengineer.com/project-management-at-big-tech/">common at Big Tech companies</a>. Autonomous teams are also the focus of the <a href="https://www.atlassian.com/agile/agile-at-scale/spotify">Spotify Model</a>, which is also used at other companies (e.g. <a href="https://www.rewe-digital.com/inside-rewe-digital/show/anna-oliver-auf-der-agile-hr-conference-2021.html">Rewe Digital</a>).</p>]]></content><author><name>Mike Klimek</name></author><category term="Tech Industry" /><summary type="html"><![CDATA[Why do traditional companies struggle with software development and what can be done about it?]]></summary></entry><entry><title type="html">Security &amp;amp; Pen Tests</title><link href="https://mike.cloud/android/2022/05/26/security.html" rel="alternate" type="text/html" title="Security &amp;amp; Pen Tests" /><published>2022-05-26T00:00:00+00:00</published><updated>2022-05-26T00:00:00+00:00</updated><id>https://mike.cloud/android/2022/05/26/security</id><content type="html" xml:base="https://mike.cloud/android/2022/05/26/security.html"><![CDATA[<p>The security of a product (backend, iOS and Android app) can be evaluated with penetration tests (often done by specialized companies). Android apps are already protected by the Android <a href="https://source.android.com/security/app-sandbox">Sandbox</a> and critical operations like authentication, authorization and content storage should be done in a secure backend, so critical findings in apps are rare. However pen tests will usually produce <strong>common findings</strong>, that are the same for most apps.</p>

<p>Management or legal might treat these common findings like a checklist, where everything must be implemented, because they are unsure of the costs and benefits and err on the side of safety or the <a href="https://en.wikipedia.org/wiki/Security_theater">feeling of security</a>. Instead advise them on a case by case basis.</p>

<p>In my experience the costs (complexity, bad UX, alienating users, false positives breaking the app) for these common findings are often high and potential benefits are low, because the OS leaves little room for improvements, so <strong>often no action is necessary</strong>. Not all apps are equal though. E.g. benefits for making it harder to analyze the app might match the costs in high value targets like banking apps.</p>

<p>Links:</p>
<ul>
  <li><a href="https://developer.android.com/topic/security/best-practices">Official Best Practices</a></li>
  <li><a href="https://developer.android.com/training/safetynet">SafetyNet</a></li>
  <li><a href="https://kenkantzer.com/learnings-from-5-years-of-tech-startup-code-audits/">Learnings from 5 years of tech startup code audits</a></li>
</ul>

<h1 id="security-tips">Security Tips</h1>
<ul>
  <li>Beware of <a href="https://en.wikipedia.org/wiki/Security_through_obscurity">security by obscurity</a>. It is incredibly easy to decompile apps or <a href="https://play.google.com/store/apps/details?id=sk.styk.martin.apkanalyzer&amp;hl=de&amp;gl=US">look at their resources</a>. It’s also possible to modify the code (remove protections) and recompile it. Consider your app’s code fully transparent and modifiable.</li>
  <li>Adversaries can run and fully analyze your app in an environment that you have no control over. Most detection heuristics are trivially circumvented. This is not an issue if your app is <a href="https://en.wikipedia.org/wiki/Secure_by_design">secure by design</a>.</li>
  <li>Let the OS handle security instead of rolling your own limited solutions. The most influential thing you can do is raising the <code class="language-plaintext highlighter-rouge">minSdkVersion</code> because old OS versions (<a href="https://www.tomsguide.com/us/old-phones-unsafe,news-24846.html">older than 3 to 5 years</a>) will not receive any security patches.</li>
  <li>Storing sensitive data in <a href="https://developer.android.com/training/data-storage/app-specific">external storage</a> used to be a critical issue, which is why it <a href="https://developer.android.com/training/data-storage/app-specific#external">was solved on OS side</a>. The app’s internal storage directory is sandboxed so it can not be accessed by other apps and is automatically encrypted since Android 10. Nevertheless it’s always a good idea to <strong>minimize persistent client side storage of sensitive data</strong>.</li>
  <li>Don’t roll your own crypto (algorithms), instead use the <a href="/android/2021/10/19/crypto.html">Android crypto APIs</a>.</li>
</ul>

<h1 id="common-findings">Common Findings</h1>
<details>
  <summary>Clear Text Traffic</summary>

  <p>This finding is likely a false positive, as with Android 9 clear text communication (not using HTTPS) is prevented by default. False positives include deeplink URLs in manifest (so app is also opened for <code class="language-plaintext highlighter-rouge">http</code> deeplinks) or support for local dev servers in debug builds.</p>

  <p>If you are actually using clear text communication, you should have a very good reason, e.g. exception in <a href="https://developer.android.com/training/articles/security-config">security config</a> for legacy backend without maintainer.</p>
</details>

<details>
  <summary>Tapjacking</summary>

  <p>Other apps may use overlays to listen for touch events (e.g. to get passwords).</p>

  <p>There are two ways this can happen:</p>
  <ul>
    <li>Overlays:
      <ul>
        <li>User has <a href="https://www.xda-developers.com/how-tapjacking-made-a-return-with-android-marshmallow-and-nobody-noticed/">Android 6.0.1 without June security patches or Android &lt;4.0.3</a>. This allowed apps to show a <code class="language-plaintext highlighter-rouge">Toast</code> (transparent or with misleading content) infront of other apps and intercept touch events.</li>
        <li>On newer OS versions the user has to explicitly give permission for apps to draw over other apps.</li>
        <li>StackOverflow solutions will recommend using <a href="https://developer.android.com/reference/kotlin/android/view/View#security">android:filterTouchesWhenObscured</a>. This will break legitimate apps like blue light filters. Instead consider raising <code class="language-plaintext highlighter-rouge">minSdkVersion</code>, because this is fixed on OS side, and these old versions also contain other security issues, like <a href="https://en.wikipedia.org/wiki/Heartbleed">Heartbleed</a> in Android 4.1.1.</li>
      </ul>
    </li>
    <li>User has a malicious keyboard app:
      <ul>
        <li>While iOS has an API for disabling third party keyboards, there is no such thing on Android.</li>
        <li>Unlike iOS there is no “first party” keyboard, because every OEM can preinstall their own keyboard app.</li>
        <li>Apart from that, I think users should be allowed to use their preferred keyboard.</li>
      </ul>
    </li>
  </ul>

  <p>You can add <a href="https://developer.android.com/reference/kotlin/android/view/WindowManager.LayoutParams#flag_secure">FLAG_SECURE</a> to <code class="language-plaintext highlighter-rouge">Window</code>s so they don’t appear in screenshots or the recent tasks preview. Preventing screenshots will annoy users, so this only makes sense in very rare cases like password managers that show plaintext passwords.</p>
</details>

<details>
  <summary>Secrets in Code</summary>

  <p>Findings will usually consider every secret (API keys, passwords, keystores, etc.) with the same severity. But not all secrets are equal. We have to consider these points:</p>
  <ul>
    <li>Does the secret have to be included in the app (APK or at runtime)?</li>
    <li>What can an adversary do with a leaked secret?</li>
    <li>Can we easily revoke and replace this secret?</li>
    <li>Is the codebase hosted in a public repository (e.g. open source project) or a private repository (proprietary project)?</li>
  </ul>

  <p>For public projects it makes sense to just hide all kinds of secrets in <a href="https://github.com/google/secrets-gradle-plugin">local.properties</a> or environment variables and provide default secrets. This makes it easier to configure forks and prevents automated bots from grabbing them. If you pushed a secret to a public repository, consider it compromised.</p>

  <p>For private projects it is preferable to save complexity and make building and deploying the app as easy as possible (<a href="https://en.wikipedia.org/wiki/Convention_over_configuration">convention over configuration</a>). Here we can distinguish between secrets that must end up in the app and those that don’t:</p>
  <ul>
    <li>Many libraries (e.g. Google Maps) store their <strong>API keys in the manifest</strong>. Users can just open the manifest and copy the keys. This is <a href="https://en.wikipedia.org/wiki/Secure_by_design">by design</a>, because there are hard restrictions on what these keys can do. As long as the key has to end up in the app at some point, e.g. if you provide these keys at runtime via a backend, it can be intercepted by a determined hacker. You can consider these kind of API keys publicly available. Having the above kind of keys <strong>in the code</strong> is fine for private code repositories. It’s probably easier to extract the key from the app, than to gain access to the codebase.</li>
    <li>For other cases, check if the secret needs to appear in the app. Maybe it is only used at compile time or for an internal test variant of the app. Maybe it can be stored in (and never leave) the backend, which then acts as a <strong>proxy to external APIs</strong>. Especially risky general purpose secrets like AWS tokens, should not end up in the app and in the code base. Be sure to also rewrite your git commit history, when removing them.</li>
  </ul>

  <p>The <strong>keystore</strong> with the key can be checked into private repositories. This is completely fine in my opinion for the following reasons:</p>
  <ul>
    <li>The keystore is useless without keystore password and key password (unless the passwords can be brute forced, in that case you should just move the key to keystore with a longer password). If an adversary also has access to the passwords, e.g. via secrets manager or CI server, they most likely also have access to other secrets like the keystore.</li>
    <li>If the keystore contains the <strong>upload key</strong> for <a href="https://developer.android.com/studio/publish/app-signing">Google Play App Signing</a>, then it would be of no use to an adversary (even if they had the passwords), as they also need access to Google Play Console (and the passwords) and you can just invalidate the upload key and generate a new one.</li>
    <li>If it contains the actual <strong>signing key</strong>, it is preferable to keep it safe in the repository, than to risk losing access to it. Adversaries still need access to Google Play Console though they might create new builds for delivery outside of Google Play if they also have access to keystore password and key password.</li>
  </ul>
</details>

<details>
  <summary>Information Leakage</summary>

  <p>If you use <a href="https://developer.android.com/studio/command-line/logcat">Logcat</a> for logging API requests, it may contain sensitive user data, tokens, etc. Other apps cannot access your app’s logs (since Android 4.1). It can still be read via <code class="language-plaintext highlighter-rouge">adb logcat</code>, however this requires direct access to the user’s device.</p>

  <p>If this is an issue, make sure that these logs do not contain sensitive data or completely remove them by using these ProGuard rules (which make debugging issues in productive apps harder):</p>

  <div class="language-conf highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># Disable logging
</span>-<span class="n">assumenosideeffects</span> <span class="n">class</span> <span class="n">android</span>.<span class="n">util</span>.<span class="n">Log</span> {
    <span class="n">public</span> <span class="n">static</span> <span class="n">boolean</span> <span class="n">isLoggable</span>(<span class="n">java</span>.<span class="n">lang</span>.<span class="n">String</span>, <span class="n">int</span>);
    <span class="n">public</span> <span class="n">static</span> <span class="n">int</span> <span class="n">v</span>(...);
    <span class="n">public</span> <span class="n">static</span> <span class="n">int</span> <span class="n">d</span>(...);
    <span class="n">public</span> <span class="n">static</span> <span class="n">int</span> <span class="n">i</span>(...);
    <span class="n">public</span> <span class="n">static</span> <span class="n">int</span> <span class="n">w</span>(...);
    <span class="n">public</span> <span class="n">static</span> <span class="n">int</span> <span class="n">e</span>(...);
}
</code></pre></div>  </div>
</details>

<details>
  <summary>No Root Detection</summary>

  <p>The app can react (e.g. stop working) to running on a rooted device, where the sandbox is not guaranteed.</p>

  <p>The benefit is small:</p>
  <ul>
    <li>It can prevent this scenario: User uses your app on a rooted device, installs a malicious app and grants it root access. The malicious app can now access your app’s internal data directory. With root detection the user could not use your app at all, so there is no data to access.</li>
    <li>Root detection is fragile and easily circumvented.</li>
  </ul>

  <p>The cost is high:</p>
  <ul>
    <li>Implementing effective root detection requires a lot of effort and complexity (see tamper protection).</li>
    <li>About <a href="https://www.verimatrix.com/knowledge-base/application-security/what-is-root-detection/">3.6%</a> of devices are rooted. Most are custom ROMs, which users install at their own risk. Some devices (One Plus, Xiaomi) are pre-rooted. Affected users can not use the app, will be frustrated and post negative reviews.</li>
  </ul>

  <p>In my opinion, this is only useful for very high risk apps (banking), if at all. In that case it could be preferable to show a message to users, that their device is not safe and they are at their own risk.</p>
</details>

<details>
  <summary>No Tamper Protection</summary>

  <p>This checks the <strong>integrity</strong> of app and environment, to find out if an adversary has recompiled the app or is running it in a <strong>hostile environment</strong> to analyze it. This usually involves:</p>
  <ul>
    <li>Root detection (see above)</li>
    <li>Emulator detection</li>
    <li>Check if debugger is connected</li>
    <li>…</li>
  </ul>

  <p>The benefit is small:</p>
  <ul>
    <li>It is marginally harder to analyze or recompile the app. Checks can be removed and circumvented.</li>
  </ul>

  <p>The cost is high:</p>
  <ul>
    <li><a href="https://developer.android.com/training/safetynet">Implementing effective tamper protection</a> requires a lot of effort and complexity. It is an arms race between protection tools and bypass tools.</li>
    <li>This might make development harder, e.g. if the app can not run on emulators anymore. Test automation might break.</li>
  </ul>

  <p>This might make sense on a <a href="https://github.com/mukeshsolanki/Android-Tamper-Detector">small scale</a>, that just checks the app certificate at runtime, to prevent automatic recompilation with injected malware.</p>
</details>

<details>
  <summary>No Obfuscation</summary>

  <p><a href="https://developer.android.com/studio/build/shrink-code">Obfuscation with R8</a> makes it harder to analyze the app, but will not stop a determined hacker. You should enable it in any case, because R8 can dramatically <strong>reduce your APK’s download size</strong> through code and resource shrinking.</p>

  <p>You have to manage a <code class="language-plaintext highlighter-rouge">proguard-rules.pro</code> file and check your release builds for shrinking issues. If this was not done from the start of the project, it <strong>might be too late</strong> to get it working, because of conflicting and really hard to debug errors. This is especially hard if you use proprietary libraries, that do not provide their own <code class="language-plaintext highlighter-rouge">consumer-rules.pro</code>.</p>

  <p>It will slow down builds, so it should only be enabled for releases:</p>
  <div class="language-groovy highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">defaultConfig</span> <span class="o">{</span>
    <span class="o">...</span>
    <span class="n">proguardFiles</span> <span class="nf">getDefaultProguardFile</span><span class="o">(</span><span class="s1">'proguard-android-optimize.txt'</span><span class="o">),</span> <span class="s1">'proguard-rules.pro'</span>
<span class="o">}</span>

<span class="n">buildTypes</span> <span class="o">{</span>
    <span class="n">debug</span> <span class="o">{</span>
        <span class="c1">// set to true, if you want to debug code shrinking:</span>
        <span class="n">minifyEnabled</span> <span class="kc">false</span>
        <span class="n">shrinkResources</span> <span class="kc">false</span>
    <span class="o">}</span>
    <span class="n">release</span> <span class="o">{</span>
        <span class="n">minifyEnabled</span> <span class="kc">true</span>
        <span class="n">shrinkResources</span> <span class="kc">true</span>
    <span class="o">}</span>
<span class="o">}</span>
</code></pre></div>  </div>
</details>

<details>
  <summary>No Certificate Pinning</summary>

  <p>Certificate pinning is not recommended by <a href="https://developer.android.com/training/articles/security-ssl#Pinning">Google</a> and 
<a href="https://developer.apple.com/news/?id=g9ejcf8y">Apple</a> and should be avoided except for regulatory requirements.</p>

  <p>The benefit is low. Communication between app and backend can not be intercepted (man-in-the-middle) in these <strong>additional special cases</strong>:</p>
  <ul>
    <li>One of the 150 certificate authorities is compromised and their certificate not immediately revoked by an OS update.</li>
    <li>Developers analyzing the app’s traffic in a hostile environment (though they can just recompile the app without certificate pinning).</li>
    <li>Only rooted devices or devices with <a href="https://blog.jeroenhd.nl/article/android-7-nougat-and-certificate-authorities">Android &lt;7</a>: Third party accessing user device’s certificate store (e.g. companies installing certificates on employee devices).</li>
  </ul>

  <p>The cost and risk is high:</p>
  <ul>
    <li>Requires considerably organizational overhead, as you need to keep certificates up to date through regular app updates.</li>
    <li>Old apps will be broken and you will get negative reviews, unless the app has a force-update mechanism.</li>
  </ul>
</details>

<details>
  <summary>Supply chain attack</summary>

  <p><a href="https://en.wikipedia.org/wiki/Supply_chain_attack">Supply chain attacks</a> are rare and usually not part of pen tests, but can be a major risk.</p>

  <p>Third party dependencies or their transitive dependencies can contain malicious code:</p>
  <ul>
    <li>Gradle plugin uploading environment variables (AWS tokens, API keys, etc.) to third party server.</li>
    <li>Library uploading user data or credentials to a third party server.</li>
    <li>Library injecting ads, bitcoin miners, etc.</li>
  </ul>

  <p>To migitate:</p>
  <ul>
    <li>Prefer common libraries:
      <ul>
        <li>Official (<code class="language-plaintext highlighter-rouge">androidx</code>, <code class="language-plaintext highlighter-rouge">com.google.*</code>, <code class="language-plaintext highlighter-rouge">org.jetbrains.*</code>) plugins and libraries are usually safe.</li>
        <li>Open source dependencies with lots of stars on GitHub are not immune to attacks but have more oversight. When updating check their changes and don’t use versions that have been out for less than a day.</li>
      </ul>
    </li>
    <li>Check source code (or even compiled byte code) of small libraries for suspicious code:
      <ul>
        <li>Unexpected HTTP requests or URLs</li>
        <li>Access to environment variables</li>
        <li>Base64 encoded strings that are decoded and executed</li>
        <li>Files with very long horizontal scrollbars due code hidden after lots of whitespace.</li>
      </ul>
    </li>
    <li>Check that you are actually using the correct library:
      <ul>
        <li>Check for typos to prevent <a href="https://en.wikipedia.org/wiki/Typosquatting">typosquatting</a> attacks.</li>
        <li>In <code class="language-plaintext highlighter-rouge">settings.gradle</code>: Ensure <code class="language-plaintext highlighter-rouge">mavenCentral()</code> and <code class="language-plaintext highlighter-rouge">google()</code> Maven repostories are defined before other repositories like <code class="language-plaintext highlighter-rouge">maven { url "https://jitpack.io" }</code>, to prevent “shadowing” of official dependencies.</li>
      </ul>
    </li>
  </ul>
</details>]]></content><author><name>Mike Klimek</name></author><category term="Android" /><summary type="html"><![CDATA[Most common Android app security findings and what to do about them.]]></summary></entry><entry><title type="html">Technical Interviews</title><link href="https://mike.cloud/tech%20industry/2022/05/22/technical-interviews.html" rel="alternate" type="text/html" title="Technical Interviews" /><published>2022-05-22T00:00:00+00:00</published><updated>2022-05-22T00:00:00+00:00</updated><id>https://mike.cloud/tech%20industry/2022/05/22/technical-interviews</id><content type="html" xml:base="https://mike.cloud/tech%20industry/2022/05/22/technical-interviews.html"><![CDATA[<p>Technical interviews have become kind of a meme in the developer community. Especially with Big Tech companies’ extensive focus on niche algorithms and datastructures whiteboard tests, that have no relevance for most developer jobs, but are still copied by non tech companies. This post contains some tips for conducting technical interviews as a developer.</p>

<h1 id="what-makes-a-good-candidate">What makes a good candidate?</h1>
<p>A good candidate is not just someone who can code or solve algorithm puzzles. You’re looking for someone who can apply these skills and <strong>produce value</strong> (directly or indirectly). For example by raising productivity and quality, satisfying customers, bringing in new knowledge, or generally making work pleasant for the team.</p>

<p>So you are looking for self motivated <strong>problem solvers</strong>, not <a href="https://en.wikipedia.org/wiki/Code_monkey">code monkeys</a>. Someone who can say “no” if there are better suited alternative solutions, that will save time and money.</p>

<p>At the end of your interview, you should be able to assess these points about a candidate:</p>
<ul>
  <li><strong>Skills</strong>: Do they already have the necessary experience to become productive? Better yet: Are they able to quickly acquire new skills and fill gaps? Missing skills is not a problem, if a candidate has high potential and is a quick learner.</li>
  <li><strong>Motivation</strong>: Can they get stuff done without constant supervision? Will they ask for help if they are stuck?  Will they proactively solve problems and strive for continuous improvement?</li>
  <li><strong>Teamwork</strong>: Can they communicate their ideas clearly and openly? Would they be a good fit for the team or project? And is the team a good fit for them?</li>
</ul>

<h1 id="conducting-technical-interviews">Conducting technical interviews</h1>

<div class="message">

  <p><strong>Note:</strong> I have only held interviews at small and medium sized companies. Processes are usually more rigid at larger coorporations. However these tips worked well in my experience.</p>
</div>

<ul>
  <li>Someone with a relevant technical background should be involved in the hiring process. Otherwise there is no way to assess technical skill levels. HR is easily fooled by just mentioning skills from the job description, even if you have only done “Hello World” projects.</li>
  <li>Remember you are talking to a human being (not a “resource”), who is in a stressful situation. Have a friendly introduction to ease the situation and so they know they are talking to a fellow developer and can get more technical. Not knowing an answer is totally fine. Don’t make them feel like they gave a wrong answer.</li>
  <li>Get them talking about their past projects, favorite libraries, worst screwups, weirdest bugs, etc. This is something they are comfortable with, shows their motivation and problem solving skills. It also reflects real development reality and is much more effective at assessing their specific <strong>strengths</strong>, than asking a fixed question catalogue and hoping the questions match up with their skills.</li>
  <li>You don’t have to talk exlusively about development topics to grasp their <strong>motivation</strong> and <strong>communication</strong> skills. You can also talk about something they are passionate about. Maybe they have some cool hobby projects. And what was their motivation to become a developer?</li>
  <li>Try to find the <strong>limits</strong> of their skills. Often there are chains of skills, that depend on each other or get more advanced with every step. For example, if they know Kotlin, ask them if and how they have used Kotlin Coroutines. If they are proficient ask them about Coroutine Flows (or RxJava). You can then talk about more advanced concepts (cold and hot streams, <a href="https://en.wikipedia.org/wiki/Unidirectional_Data_Flow_(computer_science)">unidirectional data flow</a>, etc.), but try not get caught up in specifics, instead keep a <strong>broad overview</strong> of their skillset.</li>
</ul>

<h1 id="topics-for-android-developers">Topics for Android Developers</h1>
<p>This is a list of topics you can talk about, that are part of typical Android development. They should give a rough indication of the skill level of an Android developer.</p>

<p>Most topics will be answered, by just talking about the details of candidate’s past projects. With more experienced developers, you will probably be talking about more advanced topics like architectures, CI/CD and specific Android APIs.</p>

<div class="message">

  <p><strong>Note:</strong> These questions are not intended as a qualitative measurable checklist.</p>
</div>

<ul>
  <li>Experience:
    <ul>
      <li>How long have you been programming Android apps?</li>
      <li>Other related skills (Flutter, iOS, Web, UX/UI, PM, etc.)?</li>
      <li>What projects did you work on?
        <ul>
          <li>Technologies?</li>
          <li>Team sizes?</li>
          <li>Responsibilities?</li>
          <li>Difficulties and solutions?</li>
          <li>…</li>
        </ul>
      </li>
    </ul>
  </li>
  <li>Technical knowledge:
    <ul>
      <li>Kotlin?</li>
      <li>Coroutines, Flows, RxJava?</li>
      <li>Databinding?</li>
      <li>REST, GraphQL?</li>
      <li>Bluetooth?</li>
      <li>Other Android APIs?</li>
    </ul>
  </li>
  <li>Libraries:
    <ul>
      <li>Networking (Retrofit, Apollo)?</li>
      <li>Dependency Injection (Dagger, Koin)?</li>
      <li>Jetpack Compose?</li>
      <li>Other libraries?</li>
    </ul>
  </li>
  <li>Architecture knowledge:
    <ul>
      <li>Architecture patterns (MVVM, MVC, MVI, …)?</li>
      <li>Can you explain how a pattern works, what are the advantages, etc.?</li>
      <li>Google’s <a href="https://developer.android.com/topic/architecture#recommended-app-arch">recommended architecture</a> (layers)?</li>
      <li>Could you plan and build an app’s architecture from scratch without help?</li>
    </ul>
  </li>
  <li>Workflows:
    <ul>
      <li>Git?</li>
      <li>Code Reviews?</li>
      <li>Git Flow?</li>
      <li>CI / CD?</li>
    </ul>
  </li>
</ul>]]></content><author><name>Mike Klimek</name></author><category term="Tech Industry" /><summary type="html"><![CDATA[Tips for conducting technical interviews.]]></summary></entry><entry><title type="html">Cryptography &amp;amp; Biometrics</title><link href="https://mike.cloud/android/2021/10/19/crypto.html" rel="alternate" type="text/html" title="Cryptography &amp;amp; Biometrics" /><published>2021-10-19T00:00:00+00:00</published><updated>2021-10-19T00:00:00+00:00</updated><id>https://mike.cloud/android/2021/10/19/crypto</id><content type="html" xml:base="https://mike.cloud/android/2021/10/19/crypto.html"><![CDATA[<p>Implementing cryptographic operations in Android without a library can be quite confusing, because the API is based on the ancient <a href="https://docs.oracle.com/javase/8/docs/technotes/guides/security/crypto/CryptoSpec.html">Java Cryptography Architecture</a>, which was introduced with JDK 1.1. However from a high-level point of view, encrypting and decrypting data in Android is relatively straight forward.</p>

<div class="message">

  <p><strong>Note:</strong> The <a href="https://developer.android.com/topic/security/data">Jetpack Security library</a> is an abstraction over these APIs and helps with encrypting <code class="language-plaintext highlighter-rouge">SharedPreferences</code> or files, but currently does not support biometrics.</p>
</div>

<p>Links:</p>
<ul>
  <li><a href="https://developer.android.com/guide/topics/security/cryptography">Official Documentation</a></li>
  <li><a href="https://developer.android.com/training/sign-in/biometric-auth">Biometric Auth</a></li>
</ul>

<h1 id="workflow">Workflow</h1>
<p>Encryption:</p>
<ol>
  <li>Obtain a <code class="language-plaintext highlighter-rouge">SecretKey</code>:
    <ul>
      <li>Either create a new key in the Android <code class="language-plaintext highlighter-rouge">KeyStore</code></li>
      <li>Or get a key that you created previously from Android <code class="language-plaintext highlighter-rouge">KeyStore</code></li>
    </ul>
  </li>
  <li>Create a <code class="language-plaintext highlighter-rouge">Cipher</code> and initialize it with the <code class="language-plaintext highlighter-rouge">SecretKey</code></li>
  <li>Encrypt data with the <code class="language-plaintext highlighter-rouge">Cipher</code></li>
  <li>Optional: Depending on the used transformation (block mode), you also need to store the <code class="language-plaintext highlighter-rouge">Cipher</code>’s generated initialization vector (<code class="language-plaintext highlighter-rouge">cipher.iv</code>), e.g. by concatenating it with the encrypted data.</li>
</ol>

<p>Decryption:</p>
<ol>
  <li>Get a <code class="language-plaintext highlighter-rouge">SecretKey</code> that you created previously from Android <code class="language-plaintext highlighter-rouge">KeyStore</code></li>
  <li>Create a <code class="language-plaintext highlighter-rouge">Cipher</code> and initialize it with the <code class="language-plaintext highlighter-rouge">SecretKey</code></li>
  <li>Optional: Depend on the used transformation, you will also need to provide the previously stored initialization vector to the <code class="language-plaintext highlighter-rouge">Cipher</code>.</li>
  <li>Decrypt data with the <code class="language-plaintext highlighter-rouge">Cipher</code></li>
</ol>

<h1 id="choosing-a-transformation">Choosing a transformation</h1>
<p>A “transformation” describes the cryptographic operations that should be used. Its format looks like this: <code class="language-plaintext highlighter-rouge">"$ALGORITHM/$BLOCK_MODE/$PADDING"</code>. Google recommends <a href="https://developer.android.com/guide/topics/security/cryptography#choose-algorithm">AES/GCM/NoPadding with 256-bit keys</a>.</p>

<details>
  <summary>Algorithm</summary>

  <ul>
    <li><a href="https://de.wikipedia.org/wiki/Advanced_Encryption_Standard">AES</a> is an industry standard and is available on all Android versions.</li>
    <li><a href="https://en.wikipedia.org/wiki/Data_Encryption_Standard">DES</a> is the predecessor of AES.</li>
    <li><a href="https://en.wikipedia.org/wiki/RSA_(cryptosystem)">RSA</a> is used for asymmetric (public / private key) cryptography.</li>
  </ul>
</details>

<details>
  <summary>Block mode</summary>

  <p>The algorithms encrypt data in blocks of fixed length, e.g. AES only encrypts 128 bit blocks. If the data to be encrypted is longer, multiple blocks have to be chained. This is handled by the <a href="https://en.wikipedia.org/wiki/Block_cipher_mode_of_operation">block mode</a> (ECB, CBC, CTR, GCM).</p>

  <p>Never use ECB, as as it forms <a href="https://words.filippo.io/the-ecb-penguin/">patterns</a>.</p>

  <p>In most modes, the first block has to be initialized with an initialization vector (IV). This IV is generated by the <code class="language-plaintext highlighter-rouge">Cipher</code> during encryption, so it can not accidentally be reused when encrypting other data. Block modes affect performance (i.e. encrypting large files).</p>

  <p>GCM also has an “authentication tag”, which is used to check data integrity. It is automatically appended to the encrypted data during encryption and split off during decryption. The tag has a length of 128 bits. It is possible (but not recommended) to store only 120, 112, 104 or 96 bits of the tag, e.g. if the blocks size is smaller than 128 bits.</p>
</details>

<details>
  <summary>Padding</summary>
  <p>If the last block does not fit the length of the data to be encrypted, it has to be <a href="https://en.wikipedia.org/wiki/Padding_(cryptography)">padded</a> (NoPadding, PKCS5Padding, PKCS7Padding).</p>
</details>

<details>
  <summary>Key size</summary>
  <p>AES has multiple key sizes: 128, 192, 256. All key sizes are secure and differences are <a href="https://security.stackexchange.com/questions/14068/why-most-people-use-256-bit-encryption-instead-of-128-bit/19762#19762">mostly theoretical or regulatory</a>.</p>
</details>

<h1 id="generating-secret-keys">Generating secret keys</h1>
<p>There are two ways to generate keys:</p>
<ul>
  <li>Generate one yourself and put it in the <code class="language-plaintext highlighter-rouge">KeyStore</code></li>
  <li>Let the key store generate one via <code class="language-plaintext highlighter-rouge">KeyGenParameterSpec</code></li>
</ul>

<p>Letting the Android key store handle key generation is most secure, as the actual key never leaves the <a href="https://source.android.com/security/trusty">trusted enviroment</a>. Example (<code class="language-plaintext highlighter-rouge">AES/GCM/NoPadding</code>):</p>
<div class="language-kotlin highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">private</span> <span class="k">fun</span> <span class="nf">generateKey</span><span class="p">(</span><span class="n">keyAlias</span><span class="p">:</span> <span class="nc">String</span><span class="p">)</span> <span class="p">{</span>
    <span class="kd">val</span> <span class="py">spec</span> <span class="p">=</span> <span class="nc">KeyGenParameterSpec</span><span class="p">.</span><span class="nc">Builder</span><span class="p">(</span><span class="n">keyAlias</span><span class="p">,</span> <span class="nc">PURPOSE_ENCRYPT</span> <span class="n">or</span> <span class="nc">PURPOSE_DECRYPT</span><span class="p">)</span>
        <span class="p">.</span><span class="nf">setKeySize</span><span class="p">(</span><span class="mi">256</span><span class="p">)</span>
        <span class="p">.</span><span class="nf">setBlockModes</span><span class="p">(</span><span class="nc">BLOCK_MODE_GCM</span><span class="p">)</span>
        <span class="p">.</span><span class="nf">setEncryptionPaddings</span><span class="p">(</span><span class="nc">ENCRYPTION_PADDING_NONE</span><span class="p">)</span>
        <span class="p">.</span><span class="nf">build</span><span class="p">()</span>
    <span class="c1">// use "AndroidKeyStore" as a security provider, so key is directly saved in key store</span>
    <span class="kd">val</span> <span class="py">keyGenerator</span> <span class="p">=</span> <span class="nc">KeyGenerator</span><span class="p">.</span><span class="nf">getInstance</span><span class="p">(</span><span class="nc">KEY_ALGORITHM_AES</span><span class="p">,</span> <span class="s">"AndroidKeyStore"</span><span class="p">)</span>
    <span class="n">keyGenerator</span><span class="p">.</span><span class="nf">init</span><span class="p">(</span><span class="n">spec</span><span class="p">)</span>
    <span class="n">keyGenerator</span><span class="p">.</span><span class="nf">generateKey</span><span class="p">()</span>
<span class="p">}</span>

<span class="k">private</span> <span class="k">fun</span> <span class="nf">getKey</span><span class="p">(</span><span class="n">keyAlias</span><span class="p">:</span> <span class="nc">String</span><span class="p">):</span> <span class="nc">Key</span> <span class="p">{</span>
    <span class="k">private</span> <span class="kd">val</span> <span class="py">keyStore</span> <span class="p">=</span> <span class="nc">KeyStore</span><span class="p">.</span><span class="nf">getInstance</span><span class="p">(</span><span class="s">"AndroidKeyStore"</span><span class="p">).</span><span class="nf">apply</span> <span class="p">{</span>
        <span class="nf">load</span><span class="p">(</span><span class="k">null</span><span class="p">)</span> <span class="c1">// before the keystore can be accessed it must be loaded</span>
    <span class="p">}</span>
    <span class="k">return</span> <span class="n">keyStore</span><span class="p">.</span><span class="nf">getKey</span><span class="p">(</span><span class="n">keyAlias</span><span class="p">,</span> <span class="k">null</span><span class="p">)</span>
<span class="p">}</span>
</code></pre></div></div>

<h1 id="using-a-cipher">Using a Cipher</h1>
<div class="language-kotlin highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">fun</span> <span class="nf">encrypt</span><span class="p">(</span><span class="n">plainBytes</span><span class="p">:</span> <span class="nc">ByteArray</span><span class="p">,</span> <span class="n">secretKey</span><span class="p">:</span> <span class="nc">Key</span><span class="p">):</span> <span class="nc">Pair</span><span class="p">&lt;</span><span class="nc">ByteArray</span><span class="p">,</span> <span class="nc">ByteArray</span><span class="p">&gt;</span> <span class="p">{</span>
    <span class="kd">val</span> <span class="py">cipher</span> <span class="p">=</span> <span class="nc">Cipher</span><span class="p">.</span><span class="nf">getInstance</span><span class="p">(</span><span class="s">"$ALGORITHM/$BLOCK_MODE/$PADDING"</span><span class="p">)</span> 
    <span class="n">cipher</span><span class="p">.</span><span class="nf">init</span><span class="p">(</span><span class="nc">ENCRYPT_MODE</span><span class="p">,</span> <span class="n">secretKey</span><span class="p">)</span>
    <span class="kd">val</span> <span class="py">encryptedBytes</span><span class="p">:</span> <span class="nc">ByteArray</span> <span class="p">=</span> <span class="n">cipher</span><span class="p">.</span><span class="nf">doFinal</span><span class="p">(</span><span class="n">plainBytes</span><span class="p">)</span>
    <span class="k">return</span> <span class="nc">Pair</span><span class="p">(</span><span class="n">cipher</span><span class="p">.</span><span class="n">iv</span><span class="p">,</span> <span class="n">encryptedBytes</span><span class="p">)</span>
<span class="p">}</span>

<span class="k">fun</span> <span class="nf">decrypt</span><span class="p">(</span><span class="n">encryptedBytes</span><span class="p">:</span> <span class="nc">ByteArray</span><span class="p">,</span> <span class="n">iv</span><span class="p">:</span> <span class="nc">ByteArray</span><span class="p">,</span> <span class="n">secretKey</span><span class="p">:</span> <span class="nc">Key</span><span class="p">):</span> <span class="nc">ByteArray</span> <span class="p">{</span>
    <span class="kd">val</span> <span class="py">cipher</span> <span class="p">=</span> <span class="nc">Cipher</span><span class="p">.</span><span class="nf">getInstance</span><span class="p">(</span><span class="s">"$ALGORITHM/$BLOCK_MODE/$PADDING"</span><span class="p">)</span>
    <span class="kd">val</span> <span class="py">spec</span><span class="p">:</span> <span class="nc">AlgorithmParameterSpec</span> <span class="p">=</span> <span class="k">when</span> <span class="p">(</span><span class="nc">BLOCK_MODE</span><span class="p">)</span> <span class="p">{</span>
        <span class="nc">BLOCK_MODE_GCM</span> <span class="p">-&gt;</span> <span class="nc">GCMParameterSpec</span><span class="p">(</span><span class="nc">GCM_TAG_SIZE</span><span class="p">,</span> <span class="n">iv</span><span class="p">)</span>
        <span class="k">else</span> <span class="p">-&gt;</span> <span class="nc">IvParameterSpec</span><span class="p">(</span><span class="n">iv</span><span class="p">)</span>
    <span class="p">}</span>
    <span class="n">cipher</span><span class="p">.</span><span class="nf">init</span><span class="p">(</span><span class="nc">DECRYPT_MODE</span><span class="p">,</span> <span class="n">secretKey</span><span class="p">,</span> <span class="n">spec</span><span class="p">)</span>
    <span class="k">return</span> <span class="n">cipher</span><span class="p">.</span><span class="nf">doFinal</span><span class="p">(</span><span class="n">bytes</span><span class="p">)</span>
<span class="p">}</span> 
</code></pre></div></div>

<h1 id="encoding-strings">Encoding Strings</h1>
<p>Input for encryption and decryption must be <code class="language-plaintext highlighter-rouge">ByteArray</code>s, as most algorithms work on blocks of bytes (e.g. 128 bit blocks for AES). You can use Base64 for easy conversion of <code class="language-plaintext highlighter-rouge">String</code>s:</p>
<div class="language-kotlin highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">private</span> <span class="k">fun</span> <span class="nc">Cipher</span><span class="p">.</span><span class="nf">encrypt</span><span class="p">(</span><span class="n">plainText</span><span class="p">:</span> <span class="nc">String</span><span class="p">):</span> <span class="nc">String</span> <span class="p">{</span>
    <span class="kd">val</span> <span class="py">plainTextBytes</span> <span class="p">=</span> <span class="n">plainText</span><span class="p">.</span><span class="nf">encodeToByteArray</span><span class="p">()</span>
    <span class="kd">val</span> <span class="py">cipherBytes</span> <span class="p">=</span> <span class="nf">doFinal</span><span class="p">(</span><span class="n">plainTextBytes</span><span class="p">)</span>
    <span class="k">return</span> <span class="n">cipherBytes</span><span class="p">.</span><span class="nf">encodeToBase64</span><span class="p">()</span>
<span class="p">}</span>

<span class="k">private</span> <span class="k">fun</span> <span class="nc">Cipher</span><span class="p">.</span><span class="nf">decrypt</span><span class="p">(</span><span class="n">cipherText</span><span class="p">:</span> <span class="nc">String</span><span class="p">):</span> <span class="nc">String</span> <span class="p">{</span>
    <span class="kd">val</span> <span class="py">cipherBytes</span> <span class="p">=</span> <span class="n">cipherText</span><span class="p">.</span><span class="nf">decodeFromBase64</span><span class="p">()</span>
    <span class="kd">val</span> <span class="py">plainTextBytes</span> <span class="p">=</span> <span class="nf">doFinal</span><span class="p">(</span><span class="n">cipherBytes</span><span class="p">)</span>
    <span class="k">return</span> <span class="n">plainTextBytes</span><span class="p">.</span><span class="nf">decodeToString</span><span class="p">()</span>
<span class="p">}</span>

<span class="k">private</span> <span class="k">fun</span> <span class="nc">ByteArray</span><span class="p">.</span><span class="nf">encodeToBase64</span><span class="p">():</span> <span class="nc">String</span> <span class="p">=</span> <span class="nc">Base64</span><span class="p">.</span><span class="nf">encodeToString</span><span class="p">(</span><span class="k">this</span><span class="p">,</span> <span class="nc">Base64</span><span class="p">.</span><span class="nc">DEFAULT</span><span class="p">)</span>
<span class="k">private</span> <span class="k">fun</span> <span class="nc">String</span><span class="p">.</span><span class="nf">decodeFromBase64</span><span class="p">():</span> <span class="nc">ByteArray</span> <span class="p">=</span> <span class="nc">Base64</span><span class="p">.</span><span class="nf">decode</span><span class="p">(</span><span class="k">this</span><span class="p">,</span> <span class="nc">Base64</span><span class="p">.</span><span class="nc">DEFAULT</span><span class="p">)</span>
</code></pre></div></div>

<h1 id="biometric-auth">Biometric auth</h1>
<p>You can show a biometric authentication dialog with the <a href="https://developer.android.com/training/sign-in/biometric-auth">androidx biometric library</a>. You can use this as a simple dialog with a success and error callback. It supports multiple authentication strengths (i.e. fingerprints, face unlock, swipe pattern, unlock PIN, etc.), depending on Android API level.</p>

<p>However you can also use it to protect a <code class="language-plaintext highlighter-rouge">SecretKey</code>, so user authentication is required before the key can be used for encrypting or decrypting. When using the biometric prompt like this, you can only use <code class="language-plaintext highlighter-rouge">BIOMETRIC_STRONG</code> authentication, so a fallback to non-biometric credentials (PIN) is not allowed and it will not work on older devices.</p>

<p>To use it like this there are multiple steps to complete:</p>
<ol>
  <li>Check that authentication is possible (user has fingerprints enrolled) with <code class="language-plaintext highlighter-rouge">biometricManager.canAuthenticate(BIOMETRIC_STRONG)</code></li>
  <li>Create a new <code class="language-plaintext highlighter-rouge">SecretKey</code> with <code class="language-plaintext highlighter-rouge">setUserAuthenticationRequired(true)</code>. Trying to encrypt / decrypt with this key without authentication will cause an exception.</li>
  <li>Create <code class="language-plaintext highlighter-rouge">Cipher</code> that is initialized with the <code class="language-plaintext highlighter-rouge">SecretKey</code> (and IV for decryption) and pass it to <code class="language-plaintext highlighter-rouge">biometricPrompt.authenticate(info, CryptoObject(cipher))</code></li>
  <li>In the <code class="language-plaintext highlighter-rouge">onAuthenticationSucceeded</code> callback, get the Cipher from the <code class="language-plaintext highlighter-rouge">AuthenticationResult</code>. It can now be used for encryption / decryption.</li>
</ol>

<div class="message">

  <p>If you want to require authentication only for decryption (e.g. to allow storing encrypted tokens after token refresh without user intervention) you can create a secret key yourself (call <code class="language-plaintext highlighter-rouge">KeyGenerator.getInstance(ALGORITHM).generateKey()</code>) and add the key to the key store under two alias with different <code class="language-plaintext highlighter-rouge">KeyProtection</code>:</p>
  <ul>
    <li>One alias only for encryption, which does not require authentication</li>
    <li>One alias only for decryption, which requires authentication</li>
  </ul>

  <p>Link: <a href="https://stackoverflow.com/questions/56564833/how-do-i-require-user-authentication-only-for-decryption-but-not-encryption/57634763#57634763">How do I require user authentication only for decryption but not encryption</a></p>
</div>

<div class="message">

  <p><strong>NOTE:</strong> Keys can not be exported and will be removed, when the app is uninstalled. They are not part of <code class="language-plaintext highlighter-rouge">android:allowBackup</code>.</p>
</div>

<div class="message">

  <p><strong>NOTE:</strong> If a user makes changes to the biometric settings (i.e. removes all fingerprints in system settings) the secret key will be invalidated by default. It will not be removed from the key store, but it will throw an exception, when trying to encrypt/decrypt it. In that case the only option is to remove it and create a new key.</p>
</div>

<h1 id="full-example">Full example</h1>
<div class="language-kotlin highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="cm">/**
 * We're using AES/GCM/NoPadding with 256-bit keys as
 * [recommended](https://developer.android.com/guide/topics/security/cryptography#choose-algorithm)
 * by Google.
 */</span>
<span class="k">private</span> <span class="kd">object</span> <span class="nc">CryptoSpec</span> <span class="p">{</span>
    <span class="k">const</span> <span class="kd">val</span> <span class="py">ALGORITHM</span> <span class="p">=</span> <span class="nc">KEY_ALGORITHM_AES</span>
    <span class="k">const</span> <span class="kd">val</span> <span class="py">KEY_SIZE</span> <span class="p">=</span> <span class="mi">256</span> <span class="c1">// AES-256</span>
    <span class="k">const</span> <span class="kd">val</span> <span class="py">BLOCK_MODE</span> <span class="p">=</span> <span class="nc">BLOCK_MODE_GCM</span>
    <span class="k">const</span> <span class="kd">val</span> <span class="py">PADDING</span> <span class="p">=</span> <span class="nc">ENCRYPTION_PADDING_NONE</span>
    <span class="k">const</span> <span class="kd">val</span> <span class="py">GCM_TAG_SIZE</span> <span class="p">=</span> <span class="mi">128</span> <span class="c1">// auth tag is automatically appended to the ciphertext, so we don't need to handle it</span>
    <span class="k">const</span> <span class="kd">val</span> <span class="py">IV_SEPARATOR</span> <span class="p">=</span> <span class="s">":"</span> <span class="c1">// not part of Base64, so does not require escaping</span>
    <span class="k">const</span> <span class="kd">val</span> <span class="py">KEY_STORE</span> <span class="p">=</span> <span class="s">"AndroidKeyStore"</span>
<span class="p">}</span>

<span class="cm">/**
 * Allows encrypting an decrypting arbitrary Strings.
 * The encrypted format contains the ciphertext (encrypted text) combined with the
 * generated initialization vector (IV) for later decryption.
 */</span>
<span class="kd">class</span> <span class="nc">CryptographyService</span> <span class="p">{</span>

    <span class="k">private</span> <span class="kd">val</span> <span class="py">keyStore</span>
        <span class="k">get</span><span class="p">()</span> <span class="p">=</span> <span class="nc">KeyStore</span><span class="p">.</span><span class="nf">getInstance</span><span class="p">(</span><span class="nc">KEY_STORE</span><span class="p">).</span><span class="nf">apply</span> <span class="p">{</span>
            <span class="nf">load</span><span class="p">(</span><span class="k">null</span><span class="p">)</span> <span class="c1">// before the keystore can be accessed it must be loaded</span>
        <span class="p">}</span>

    <span class="k">fun</span> <span class="nf">encrypt</span><span class="p">(</span><span class="n">plainText</span><span class="p">:</span> <span class="nc">String</span><span class="p">,</span> <span class="n">keyAlias</span><span class="p">:</span> <span class="nc">String</span><span class="p">):</span> <span class="nc">String</span> <span class="p">{</span>
        <span class="k">try</span> <span class="p">{</span>
            <span class="kd">val</span> <span class="py">secretKey</span> <span class="p">=</span> <span class="n">keyStore</span><span class="p">.</span><span class="nf">getOrCreateSecretKey</span><span class="p">(</span><span class="n">keyAlias</span><span class="p">)</span>
            <span class="kd">val</span> <span class="py">cipher</span> <span class="p">=</span> <span class="nf">createCipher</span><span class="p">(</span><span class="nc">ENCRYPT_MODE</span><span class="p">,</span> <span class="n">secretKey</span><span class="p">)</span>
            <span class="k">return</span> <span class="n">cipher</span><span class="p">.</span><span class="nf">encrypt</span><span class="p">(</span><span class="n">plainText</span><span class="p">)</span>
        <span class="p">}</span> <span class="k">catch</span> <span class="p">(</span><span class="n">e</span><span class="p">:</span> <span class="nc">Exception</span><span class="p">)</span> <span class="p">{</span> <span class="c1">// e.g. NoSuchAlgorithmException, etc.</span>
            <span class="n">keyStore</span><span class="p">.</span><span class="nf">reset</span><span class="p">(</span><span class="n">keyAlias</span><span class="p">)</span>
            <span class="k">throw</span> <span class="n">e</span>
        <span class="p">}</span>
    <span class="p">}</span>

    <span class="k">fun</span> <span class="nf">decrypt</span><span class="p">(</span><span class="n">ivAndCipherText</span><span class="p">:</span> <span class="nc">String</span><span class="p">,</span> <span class="n">keyAlias</span><span class="p">:</span> <span class="nc">String</span><span class="p">):</span> <span class="nc">String</span> <span class="p">{</span>
        <span class="k">try</span> <span class="p">{</span>
            <span class="kd">val</span> <span class="py">secretKey</span> <span class="p">=</span> <span class="n">keyStore</span><span class="p">.</span><span class="nf">getSecretKey</span><span class="p">(</span><span class="n">keyAlias</span><span class="p">)</span> <span class="o">?:</span> <span class="k">throw</span> <span class="nc">CryptoException</span><span class="p">.</span><span class="nc">KeyNotFound</span>
            <span class="kd">val</span> <span class="p">(</span><span class="py">ivSpec</span><span class="p">,</span> <span class="py">cipherText</span><span class="p">)</span> <span class="p">=</span> <span class="n">ivAndCipherText</span><span class="p">.</span><span class="nf">splitIvAndCipherText</span><span class="p">()</span>
            <span class="kd">val</span> <span class="py">cipher</span> <span class="p">=</span> <span class="nf">createCipher</span><span class="p">(</span><span class="nc">DECRYPT_MODE</span><span class="p">,</span> <span class="n">secretKey</span><span class="p">,</span> <span class="n">ivSpec</span><span class="p">)</span>
            <span class="k">return</span> <span class="n">cipher</span><span class="p">.</span><span class="nf">decrypt</span><span class="p">(</span><span class="n">cipherText</span><span class="p">)</span>
        <span class="p">}</span> <span class="k">catch</span> <span class="p">(</span><span class="n">e</span><span class="p">:</span> <span class="nc">Exception</span><span class="p">)</span> <span class="p">{</span> <span class="c1">// e.g. KeyPermanentlyInvalidatedException</span>
            <span class="n">keyStore</span><span class="p">.</span><span class="nf">reset</span><span class="p">(</span><span class="n">keyAlias</span><span class="p">)</span>
            <span class="k">throw</span> <span class="n">e</span>
        <span class="p">}</span>
    <span class="p">}</span>
<span class="p">}</span>

<span class="k">private</span> <span class="k">fun</span> <span class="nf">createCipher</span><span class="p">(</span><span class="n">opmode</span><span class="p">:</span> <span class="nc">Int</span><span class="p">,</span> <span class="n">key</span><span class="p">:</span> <span class="nc">Key</span><span class="p">,</span> <span class="n">params</span><span class="p">:</span> <span class="nc">AlgorithmParameterSpec</span><span class="p">?</span> <span class="p">=</span> <span class="k">null</span><span class="p">)</span> <span class="p">=</span> <span class="nc">Cipher</span>
    <span class="p">.</span><span class="nf">getInstance</span><span class="p">(</span><span class="s">"$ALGORITHM/$BLOCK_MODE/$PADDING"</span><span class="p">)</span>
    <span class="p">.</span><span class="nf">apply</span> <span class="p">{</span> <span class="k">if</span> <span class="p">(</span><span class="n">params</span> <span class="p">!=</span> <span class="k">null</span><span class="p">)</span> <span class="nf">init</span><span class="p">(</span><span class="n">opmode</span><span class="p">,</span> <span class="n">key</span><span class="p">,</span> <span class="n">params</span><span class="p">)</span> <span class="k">else</span> <span class="nf">init</span><span class="p">(</span><span class="n">opmode</span><span class="p">,</span> <span class="n">key</span><span class="p">)</span> <span class="p">}</span>

<span class="k">private</span> <span class="k">fun</span> <span class="nc">KeyStore</span><span class="p">.</span><span class="nf">getOrCreateSecretKey</span><span class="p">(</span><span class="n">keyAlias</span><span class="p">:</span> <span class="nc">String</span><span class="p">):</span> <span class="nc">Key</span> <span class="p">{</span>
    <span class="kd">var</span> <span class="py">secretKey</span> <span class="p">=</span> <span class="nf">getSecretKey</span><span class="p">(</span><span class="n">keyAlias</span><span class="p">)</span>
    <span class="k">if</span> <span class="p">(</span><span class="n">secretKey</span> <span class="p">==</span> <span class="k">null</span><span class="p">)</span> <span class="p">{</span>
        <span class="nf">generateSecretKey</span><span class="p">(</span><span class="n">keyAlias</span><span class="p">)</span>
        <span class="n">secretKey</span> <span class="p">=</span> <span class="nf">getSecretKey</span><span class="p">(</span><span class="n">keyAlias</span><span class="p">)</span>
    <span class="p">}</span>
    <span class="k">return</span> <span class="n">secretKey</span> <span class="o">?:</span> <span class="k">throw</span> <span class="nc">CryptoException</span><span class="p">.</span><span class="nc">KeyNotFound</span>
<span class="p">}</span>

<span class="k">private</span> <span class="k">fun</span> <span class="nc">KeyStore</span><span class="p">.</span><span class="nf">getSecretKey</span><span class="p">(</span><span class="n">keyAlias</span><span class="p">:</span> <span class="nc">String</span><span class="p">)</span> <span class="p">=</span> <span class="nf">getKey</span><span class="p">(</span><span class="n">keyAlias</span><span class="p">,</span> <span class="k">null</span><span class="p">)</span>

<span class="k">private</span> <span class="k">fun</span> <span class="nc">KeyStore</span><span class="p">.</span><span class="nf">reset</span><span class="p">(</span><span class="n">keyAlias</span><span class="p">:</span> <span class="nc">String</span><span class="p">)</span> <span class="p">=</span> <span class="nf">runCatching</span> <span class="p">{</span> <span class="nf">deleteEntry</span><span class="p">(</span><span class="n">keyAlias</span><span class="p">)</span> <span class="p">}</span>

<span class="k">private</span> <span class="k">fun</span> <span class="nc">KeyStore</span><span class="p">.</span><span class="nf">generateSecretKey</span><span class="p">(</span><span class="n">keyAlias</span><span class="p">:</span> <span class="nc">String</span><span class="p">)</span> <span class="p">{</span>
    <span class="kd">val</span> <span class="py">spec</span> <span class="p">=</span> <span class="nc">KeyGenParameterSpec</span><span class="p">.</span><span class="nc">Builder</span><span class="p">(</span><span class="n">keyAlias</span><span class="p">,</span> <span class="nc">PURPOSE_ENCRYPT</span> <span class="n">or</span> <span class="nc">PURPOSE_DECRYPT</span><span class="p">)</span>
        <span class="p">.</span><span class="nf">setKeySize</span><span class="p">(</span><span class="nc">KEY_SIZE</span><span class="p">)</span>
        <span class="p">.</span><span class="nf">setBlockModes</span><span class="p">(</span><span class="nc">BLOCK_MODE</span><span class="p">)</span>
        <span class="p">.</span><span class="nf">setEncryptionPaddings</span><span class="p">(</span><span class="nc">PADDING</span><span class="p">)</span>
        <span class="p">.</span><span class="nf">build</span><span class="p">()</span>
    <span class="kd">val</span> <span class="py">keyGenerator</span> <span class="p">=</span> <span class="nc">KeyGenerator</span>
        <span class="p">.</span><span class="nf">getInstance</span><span class="p">(</span><span class="nc">ALGORITHM</span><span class="p">,</span> <span class="nc">KEY_STORE</span><span class="p">)</span>
        <span class="p">.</span><span class="nf">apply</span> <span class="p">{</span> <span class="nf">init</span><span class="p">(</span><span class="n">spec</span><span class="p">)</span> <span class="p">}</span>
    <span class="n">keyGenerator</span><span class="p">.</span><span class="nf">generateKey</span><span class="p">()</span>
<span class="p">}</span>

<span class="k">private</span> <span class="k">fun</span> <span class="nc">String</span><span class="p">.</span><span class="nf">splitIvAndCipherText</span><span class="p">():</span> <span class="nc">Pair</span><span class="p">&lt;</span><span class="nc">AlgorithmParameterSpec</span><span class="p">,</span> <span class="nc">String</span><span class="p">&gt;</span> <span class="p">{</span>
    <span class="kd">val</span> <span class="p">(</span><span class="py">ivString</span><span class="p">,</span> <span class="py">cipherText</span><span class="p">)</span> <span class="p">=</span> <span class="nf">split</span><span class="p">(</span><span class="nc">IV_SEPARATOR</span><span class="p">)</span>
    <span class="kd">val</span> <span class="py">ivSpec</span> <span class="p">=</span> <span class="nc">GCMParameterSpec</span><span class="p">(</span><span class="nc">GCM_TAG_SIZE</span><span class="p">,</span> <span class="n">ivString</span><span class="p">.</span><span class="nf">decodeFromBase64</span><span class="p">())</span>
    <span class="k">return</span> <span class="nc">Pair</span><span class="p">(</span><span class="n">ivSpec</span><span class="p">,</span> <span class="n">cipherText</span><span class="p">)</span>
<span class="p">}</span>

<span class="k">private</span> <span class="k">fun</span> <span class="nc">Cipher</span><span class="p">.</span><span class="nf">encrypt</span><span class="p">(</span><span class="n">plainText</span><span class="p">:</span> <span class="nc">String</span><span class="p">):</span> <span class="nc">String</span> <span class="p">{</span>
    <span class="kd">val</span> <span class="py">ivString</span> <span class="p">=</span> <span class="n">iv</span><span class="p">.</span><span class="nf">encodeToBase64</span><span class="p">()</span>
    <span class="kd">val</span> <span class="py">plainTextBytes</span> <span class="p">=</span> <span class="n">plainText</span><span class="p">.</span><span class="nf">encodeToByteArray</span><span class="p">()</span>
    <span class="kd">val</span> <span class="py">cipherBytes</span> <span class="p">=</span> <span class="nf">doFinal</span><span class="p">(</span><span class="n">plainTextBytes</span><span class="p">)</span>
    <span class="kd">val</span> <span class="py">cipherText</span> <span class="p">=</span> <span class="n">cipherBytes</span><span class="p">.</span><span class="nf">encodeToBase64</span><span class="p">()</span>
    <span class="k">return</span> <span class="s">"$ivString$IV_SEPARATOR$cipherText"</span>
<span class="p">}</span>

<span class="k">private</span> <span class="k">fun</span> <span class="nc">Cipher</span><span class="p">.</span><span class="nf">decrypt</span><span class="p">(</span><span class="n">cipherText</span><span class="p">:</span> <span class="nc">String</span><span class="p">):</span> <span class="nc">String</span> <span class="p">{</span>
    <span class="kd">val</span> <span class="py">cipherBytes</span> <span class="p">=</span> <span class="n">cipherText</span><span class="p">.</span><span class="nf">decodeFromBase64</span><span class="p">()</span>
    <span class="kd">val</span> <span class="py">plainTextBytes</span> <span class="p">=</span> <span class="nf">doFinal</span><span class="p">(</span><span class="n">cipherBytes</span><span class="p">)</span>
    <span class="k">return</span> <span class="n">plainTextBytes</span><span class="p">.</span><span class="nf">decodeToString</span><span class="p">()</span>
<span class="p">}</span>

<span class="k">private</span> <span class="k">fun</span> <span class="nc">ByteArray</span><span class="p">.</span><span class="nf">encodeToBase64</span><span class="p">():</span> <span class="nc">String</span> <span class="p">=</span> <span class="nc">Base64</span><span class="p">.</span><span class="nf">encodeToString</span><span class="p">(</span><span class="k">this</span><span class="p">,</span> <span class="nc">Base64</span><span class="p">.</span><span class="nc">DEFAULT</span><span class="p">)</span>
<span class="k">private</span> <span class="k">fun</span> <span class="nc">String</span><span class="p">.</span><span class="nf">decodeFromBase64</span><span class="p">():</span> <span class="nc">ByteArray</span> <span class="p">=</span> <span class="nc">Base64</span><span class="p">.</span><span class="nf">decode</span><span class="p">(</span><span class="k">this</span><span class="p">,</span> <span class="nc">Base64</span><span class="p">.</span><span class="nc">DEFAULT</span><span class="p">)</span>
</code></pre></div></div>]]></content><author><name>Mike Klimek</name></author><category term="Android" /><summary type="html"><![CDATA[Learn how to securely encrypt data using the Android key store.]]></summary></entry><entry><title type="html">Bluetooth</title><link href="https://mike.cloud/android/2021/05/19/bluetooth.html" rel="alternate" type="text/html" title="Bluetooth" /><published>2021-05-19T00:00:00+00:00</published><updated>2021-05-19T00:00:00+00:00</updated><id>https://mike.cloud/android/2021/05/19/bluetooth</id><content type="html" xml:base="https://mike.cloud/android/2021/05/19/bluetooth.html"><![CDATA[<p>Developing Bluetooth apps can be tricky, to put it mildly. This guide is based on my experience working on Bluetooth (LE and Classic) companion apps for devices like headsets, scooters, cars and smartwatches.</p>

<div class="message">

  <p><strong>WARNING:</strong> The Android Bluetooth LE API has lots of hidden traps and fragmentation issues. To save yourself some headache consider using a thoroughly tested <a href="#libraries">library</a>.</p>
</div>

<p>Links:</p>
<ul>
  <li><a href="https://developer.android.com/guide/topics/connectivity/bluetooth">Official Documentation</a></li>
  <li><a href="https://punchthrough.com/android-ble-guide/">The Ultimate Guide to Android Bluetooth Low Energy</a></li>
  <li><a href="https://www.hellsoft.se/bluetooth-le-for-modern-android-development-part-1/">Bluetooth LE for modern Android Development</a></li>
  <li><a href="https://github.com/oesmith/gatt-xml">Bluetooth LE Standard Services and Characteristics</a></li>
</ul>

<h1 id="standards">Standards</h1>
<p>There are two Bluetooth standards:</p>
<ul>
  <li><strong>Bluetooth Classic</strong>: Used for high data throughput like file and audio transfer. Uses a two-way <a href="https://developer.android.com/reference/kotlin/android/bluetooth/BluetoothSocket">socket</a> for communication. Supports <a href="https://developer.android.com/guide/topics/connectivity/bluetooth/profiles">standardized profiles</a>, like headset (volume control), A2DP (audio transfer) or health devices (step sensors). Sometimes called BR/EDR (Bluetooth Basic Rate/Enhanced Data Rate) or RFCOMM.</li>
  <li><strong>Bluetooth Low Energy</strong> (BLE): Used for low throughput interaction with IoT devices. Endpoints are called characteristics (e.g. “firmware revision”) and are grouped into services (e.g. “device information”). There are lots of <a href="https://www.bluetooth.com/specifications/specs/">standardized services and characteristics</a>, like “battery service” or “device information” service. Sometimes called GATT (Generic Attribute Profile).</li>
</ul>

<p>The Android Bluetooth API is a partial mix of both standards. E.g. Bluetooth Classic and BLE devices are both represented by the <a href="https://developer.android.com/reference/android/bluetooth/BluetoothDevice">BluetoothDevice</a> class (with <code class="language-plaintext highlighter-rouge">DEVICE_TYPE_CLASSIC</code>, <code class="language-plaintext highlighter-rouge">DEVICE_TYPE_LE</code> or <code class="language-plaintext highlighter-rouge">DEVICE_TYPE_DUAL</code>). Scanning and communication however use entirely different classes and methods.</p>

<h1 id="preconditions">Preconditions</h1>
<p>The following preconditions are required for <strong>scanning</strong> Bluetooth Classic or BLE devices:</p>
<ul>
  <li>Permission:
    <ul>
      <li>API &lt; 29 → <code class="language-plaintext highlighter-rouge">BLUETOOTH</code>, <code class="language-plaintext highlighter-rouge">BLUETOOTH_ADMIN</code>, <code class="language-plaintext highlighter-rouge">ACCESS_COARSE_LOCATION</code></li>
      <li>API ≥ 29 → <code class="language-plaintext highlighter-rouge">BLUETOOTH</code>, <code class="language-plaintext highlighter-rouge">BLUETOOTH_ADMIN</code>, <code class="language-plaintext highlighter-rouge">ACCESS_FINE_LOCATION</code> (runtime permission),<code class="language-plaintext highlighter-rouge">ACCESS_BACKGROUND_LOCATION</code> (only for scanning in background service)</li>
      <li>API ≥ 31 → <code class="language-plaintext highlighter-rouge">BLUETOOTH_SCAN</code> (runtime permission) + <code class="language-plaintext highlighter-rouge">neverForLocation</code> Flag</li>
    </ul>
  </li>
  <li><a href="https://developer.android.com/reference/kotlin/android/bluetooth/BluetoothAdapter#getstate">Bluetooth is on</a></li>
  <li>API &lt; 31 → <a href="https://developer.android.com/reference/kotlin/android/location/LocationManager#islocationenabled">location service</a> (“GPS”) is on</li>
</ul>

<p>The following preconditions are required for <strong>connecting</strong> to Bluetooth Classic or BLE devices:</p>
<ul>
  <li>Permission:
    <ul>
      <li>API &lt; 31 → <code class="language-plaintext highlighter-rouge">BLUETOOTH</code></li>
      <li>API ≥ 31 → <code class="language-plaintext highlighter-rouge">BLUETOOTH_CONNECT</code> (runtime permission)</li>
    </ul>
  </li>
  <li><a href="https://developer.android.com/reference/kotlin/android/bluetooth/BluetoothAdapter#getstate">Bluetooth is on</a></li>
  <li>Reference to a <a href="https://developer.android.com/reference/kotlin/android/bluetooth/BluetoothDevice">BluetoothDevice</a> instance.</li>
</ul>

<p>These preconditions <strong>can be lost at any time</strong>:</p>
<ul>
  <li>Bluetooth and location service can be turned off via quick settings even while the app is in foreground.</li>
  <li>Location permission can be removed in system settings or if app is running in background.</li>
  <li>Devices can be removed in system settings.</li>
</ul>

<h1 id="scanning">Scanning</h1>
<p>If the device is already bonded (i.e. it is in the list of paired device in system settings), it can be access via <a href="https://developer.android.com/reference/android/bluetooth/BluetoothAdapter#getBondedDevices()">BluetoothAdapter.getBondedDevices</a> and <strong>no scanning is needed</strong>.</p>

<h2 id="bluetooth-classic">Bluetooth Classic</h2>
<p>You can scan for nearby Bluetooth Classic devices by calling <a href="https://developer.android.com/reference/android/bluetooth/BluetoothAdapter#startDiscovery()">BluetoothAdapter.startDiscovery</a>. The <code class="language-plaintext highlighter-rouge">BluetoothAdapter</code> is a system service, that is shared by all apps, so results are passed via broadcast receivers:</p>
<ul>
  <li><code class="language-plaintext highlighter-rouge">ACTION_DISCOVERY_STARTED</code>/<code class="language-plaintext highlighter-rouge">ACTION_DISCOVERY_FINISHED</code>: Discovery started / stopped</li>
  <li><code class="language-plaintext highlighter-rouge">ACTION_FOUND</code>: Device found</li>
</ul>

<p>Discovery will stop itself after some time and needs to be restarted manually if you want continuous discovery.</p>

<h2 id="bluetooth-le">Bluetooth LE</h2>
<p>Some libraries support scanning. If not, you can scan for nearby Bluetooth LE devices by calling <a href="https://developer.android.com/reference/kotlin/android/bluetooth/le/BluetoothLeScanner#startscan">BluetoothLeScanner.startScan</a>. Be prepared for undocumented internal scan limits and irrecoverable states in the bluetooth adapter, that may require a reboot on old devices.</p>

<p>BLE scans happen in two steps:</p>
<ol>
  <li>The smartphone passively listens for advertising packages that BLE devices periodically broadcast.</li>
  <li>Once a device is found, the smartphone actively requests additional “scan response” data from it.</li>
</ol>

<p>Every BLE service can have some arbitrary advertising data. This data can be accessed via <a href="https://developer.android.com/reference/kotlin/android/bluetooth/le/ScanRecord#getservicedata">ScanRecord.getServiceData</a>.</p>

<div class="message">

  <p>Parsing of the <code class="language-plaintext highlighter-rouge">ScanRecord</code> data may be broken in some cases in Android &lt; 8. In that case we still have access to the raw data via <a href="https://developer.android.com/reference/kotlin/android/bluetooth/le/ScanRecord#getbytes">ScanRecord.getBytes</a>, so we can <a href="https://github.com/dariuszseweryn/RxAndroidBle/blob/master/rxandroidble/src/main/java/com/polidea/rxandroidble2/internal/util/ScanRecordParser.java">parse them manually</a>.</p>
</div>

<h1 id="connecting">Connecting</h1>
<p>Once you have a <a href="https://developer.android.com/reference/kotlin/android/bluetooth/BluetoothDevice">BluetoothDevice</a> (e.g. from <a href="https://developer.android.com/reference/kotlin/android/bluetooth/le/ScanResult#getdevice">ScanResult.getDevice</a>, <a href="https://developer.android.com/reference/android/bluetooth/BluetoothAdapter#getBondedDevices()">BluetoothAdapter.getBondedDevices</a>, etc.), you can connect to it.</p>

<h2 id="bluetooth-classic-1">Bluetooth Classic</h2>
<p>Calling <a href="https://developer.android.com/reference/kotlin/android/bluetooth/BluetoothDevice#createRfcommSocketToServiceRecord(java.util.UUID)">BluetoothDevice.createRfcommSocketToServiceRecord</a> will create a <a href="https://developer.android.com/reference/kotlin/android/bluetooth/BluetoothSocket">BluetoothSocket</a>. The service record UUID parameter depends on the device, but it is often the well known SPP-UUID (<code class="language-plaintext highlighter-rouge">00001101-0000-1000-8000-00805F9B34FB</code>). The device’s supported service record UUIDs can be accessed with <a href="https://developer.android.com/reference/kotlin/android/bluetooth/BluetoothDevice#getuuids">BluetoothDevice.getUuids</a>.</p>

<p>Calling <a href="https://developer.android.com/reference/kotlin/android/bluetooth/BluetoothSocket#connect">BluetoothSocket.connect</a> will block until the device is connected. This is usually just takes a few milliseconds, if the device is in range. You can then use the socket’s threadsafe <a href="https://developer.android.com/reference/kotlin/android/bluetooth/BluetoothSocket#getinputstream">input</a> and <a href="https://developer.android.com/reference/kotlin/android/bluetooth/BluetoothSocket#getoutputstream">output</a> streams, which can be used to send and retrieve arbitrary bytes.</p>

<h2 id="bluetooth-le-1">Bluetooth LE</h2>
<p>Most libraries support connecting. If not, you can connect to a Bluetooth LE device by calling <a href="https://developer.android.com/reference/kotlin/android/bluetooth/BluetoothDevice#connectgatt_3">BluetoothDevice.connectGatt</a>. The behavior of the <code class="language-plaintext highlighter-rouge">autoConnect</code> flag is poorly documented and depends on the smartphone manufacturer and Android version. In my experience it’s easier to just not use it and build your own retry mechanism instead.</p>

<h1 id="bonding">Bonding</h1>
<p>Pairing and bonding are actually two related concepts, however Android makes no distinction:</p>
<ul>
  <li>Pairing: Exchange of cryptographic keys for encrypted communication, i.e. to prevent MITM attacks.</li>
  <li>Bonding: Long term storage of encrypted keys, instead of discarding them after disconnecting.</li>
</ul>

<p>A bonded device will appear in the list of bluetooth devices in the system settings and it can be access via <a href="https://developer.android.com/reference/android/bluetooth/BluetoothAdapter#getBondedDevices()">BluetoothAdapter.getBondedDevices</a>.</p>

<p>Bluetooth Classic always requires pairing for communication. BLE requires pairing for communicating with <strong>protected characteristics</strong> (encryption required). For IoT devices without protected BLE characteristics, no pairing is required, so you can just scan for devices and read their values.</p>

<p>There are multiple ways to trigger bonding:</p>
<ul>
  <li>A user can bond Bluetooth devices in system settings, though some smartphones do not show Bluetooth-LE device there, depending on Android versions or manufacturer.</li>
  <li>Calling <a href="https://developer.android.com/reference/kotlin/android/bluetooth/BluetoothDevice#createBond()">BluetoothDevice.createBond</a>, which does pretty much the same as clicking on a device in system settings.</li>
  <li>BLE only:
    <ul>
      <li>Reading a protected characteristic, while the device is not bonded yet, will fail with an error (<code class="language-plaintext highlighter-rouge">GATT_INSUFFICIENT_AUTHENTICATION</code>) and trigger bonding.</li>
      <li>The device can trigger bonding after connecting, though this will show a pairing dialog every time, even if the device is already bonded.</li>
    </ul>
  </li>
</ul>

<p>Pairing takes a few seconds and sometimes randomly fails, so there has to be a retry mechanism. Results are passed via broadcast receivers:</p>
<ul>
  <li><code class="language-plaintext highlighter-rouge">ACTION_BOND_STATE_CHANGED</code>: Device started pairing, device was bonded, bonding failed (<a href="https://developer.android.com/reference/kotlin/android/bluetooth/BluetoothDevice#bond_none">BOND_NONE</a>), or device lost bonding (“forget device” in system settings).</li>
  <li><code class="language-plaintext highlighter-rouge">ACTION_PAIRING_REQUEST</code>: Dialog (see below) is shown, so user can confirm pairing, e.g. to <a href="https://developer.android.com/reference/kotlin/android/bluetooth/BluetoothDevice#setpin">enter a PIN</a> or authorized access to contacts.</li>
</ul>

<p><img src="https://i.stack.imgur.com/p9BNq.png" alt="Pairing request" /></p>

<h1 id="communication">Communication</h1>
<h2 id="bluetooth-classic-2">Bluetooth Classic</h2>
<p>Bluetooth Classic uses a simple duplex <a href="https://developer.android.com/reference/kotlin/android/bluetooth/BluetoothSocket">BluetoothSocket</a> for communication. It has an input and output stream, so you can write and receive arbitrary bytes to and from the device.</p>

<p>This socket is not used for audio transfer (<a href="https://developer.android.com/reference/kotlin/android/bluetooth/BluetoothA2dp">A2DP</a> profile), but for custom communication or control protocols.</p>

<p>There is no global standard for how these bytes are formatted, so a custom protocol has to be specified. This is usually a packet frame structure with header and payload.</p>

<h2 id="bluetooth-le-2">Bluetooth LE</h2>
<div class="message">

  <p><strong>WARNING:</strong> This is a tricky part. Use a <a href="#libraries">library</a> to make your life easier.</p>
</div>

<p>When connecting with <a href="https://developer.android.com/reference/kotlin/android/bluetooth/BluetoothDevice#connectgatt">BluetoothDevice.connectGatt</a>, you have to provide a <a href="https://developer.android.com/reference/kotlin/android/bluetooth/BluetoothGattCallback">BluetoothGattCallback</a>, and will get a <a href="https://developer.android.com/reference/kotlin/android/bluetooth/BluetoothGatt">BluetoothGatt</a> object.</p>

<p><a href="https://developer.android.com/reference/kotlin/android/bluetooth/BluetoothGattCallback">BluetoothGattCallback</a> is a “god callback”, because every unrelated action (characteristic reads, writes, MTU changes, etc.) is piped through it and has to be manually associated with the triggering request.</p>

<p><a href="https://developer.android.com/reference/kotlin/android/bluetooth/BluetoothGatt">BluetoothGatt</a> is used for every interaction with the device (send requests, start discovery, start MTU negotiation) and also handles (re-)connection, disconnection and resource cleanup. For reconnection it is usually better to just completely <code class="language-plaintext highlighter-rouge">close</code> the <code class="language-plaintext highlighter-rouge">BluetoothGatt</code>, remove any <code class="language-plaintext highlighter-rouge">BluetoothGatt</code> references and call <a href="https://developer.android.com/reference/kotlin/android/bluetooth/BluetoothDevice#connectgatt">BluetoothDevice.connectGatt</a> again after a short delay.</p>

<p>The Android Bluetooth LE API does not make it obvious, that you need to write your own scheduler and job queue for scheduling requests:</p>
<ul>
  <li>For every request on the <code class="language-plaintext highlighter-rouge">BluetoothGatt</code> object, you have to wait for a response on the <code class="language-plaintext highlighter-rouge">BluetoothGattCallback</code>. Calling <code class="language-plaintext highlighter-rouge">BluetoothGatt.readCharacteristic</code> before the <code class="language-plaintext highlighter-rouge">BluetoothGattCallback</code> is called from the previous request, will lead to irrecoverable states.</li>
  <li>Some comments on StackOverflow will recommend adding <code class="language-plaintext highlighter-rouge">Thead.sleep</code> to solve this problem. Do not do that. Blocking the callback thread may crash the Bluetooth stack on some smartphones, which requires a full reboot, unless you provided a <code class="language-plaintext highlighter-rouge">handler</code> to <code class="language-plaintext highlighter-rouge">BluetoothDevice.connectGatt</code> on API ≥ 26.</li>
</ul>

<p>There will be undocumented, obscure and platform dependent error codes.</p>

<h1 id="architecture">Architecture</h1>
<p>There is no best solution for an Android app’s Bluetooth architecture, as it often depends on your use case, but it’s usually a good idea to split it into separate business logic components:</p>
<ul>
  <li><code class="language-plaintext highlighter-rouge">PreconditionResolver</code>: A component for solving preconditions and providing the preconditions state.</li>
  <li><code class="language-plaintext highlighter-rouge">Scanner</code>: A component for scanning for nearby devices.</li>
  <li><code class="language-plaintext highlighter-rouge">BondingService</code>: A component for bonding to a device and providing the list of bonded devices.</li>
  <li><code class="language-plaintext highlighter-rouge">DeviceProvider</code>: A component that provides a device for connecting. It does this by remembering a unique identifier (e.g. <a href="https://developer.android.com/reference/kotlin/android/bluetooth/BluetoothDevice#getaddress">MAC-Address</a>) of the last selected device. E.g. if the user can own multiple similar devices, that are all bonded with the smartphone, then the app reconnects to last selected device instead of a random one.</li>
  <li><code class="language-plaintext highlighter-rouge">AutoConnector</code>: A component for automatically (re-)connecting to a device.
    <ul>
      <li>This component likely has a <code class="language-plaintext highlighter-rouge">start</code> and <code class="language-plaintext highlighter-rouge">stop</code> method, which control its lifecycle. Lifecycle can be controlled by the application layer, e.g. “try to connect as long as the dashboard is visible”.</li>
      <li>This component usually contains a state machine, that is run every time preconditions change, and aggressively tries to establish and maintain a connection (lots of retry logic) once <code class="language-plaintext highlighter-rouge">start</code> is called. E.g. once <code class="language-plaintext highlighter-rouge">start</code> is called, it should automatically connect to a paired device, the moment all preconditions are resolved.</li>
      <li>Once connected, it provides an abstract <code class="language-plaintext highlighter-rouge">Connection</code>, that can be used to communicate with a device. It should not contain domain specific logic like specific characteristics or command, just a way to communicate with the device.</li>
    </ul>
  </li>
  <li>Multiple domain specific components for communication, that use the <code class="language-plaintext highlighter-rouge">AutoConnector</code>’s <code class="language-plaintext highlighter-rouge">Connection</code> to send messages to the device. E.g. an extension function on the <code class="language-plaintext highlighter-rouge">Connection</code> for sending a read request to the “battery level” characteristic.</li>
</ul>

<p>The view layer should be <a href="https://en.wikipedia.org/wiki/Fundamental_theorem_of_software_engineering">separated</a> as far as possible from the Bluetooth stack. Bluetooth implementation details should not leak through the architecture stack. E.g. a <code class="language-plaintext highlighter-rouge">ViewModel</code> should not even know, that it is sending a command via Bluetooth LE or how requests are encoded on the byte level.</p>

<h1 id="other-topics">Other Topics</h1>
<h2 id="bluetooth-le-mtu">Bluetooth LE MTU</h2>
<p>The Maximum Transmission Unit (MTU) is the size of a Bluetooth LE packet. The minimum size is 23, which leaves us with <strong>20 bytes of usable payload</strong>. Trying to send or receive larger packets than the MTU will fail.</p>

<p>However smartphones and Bluetooth 4.2 devices can negotiate a larger MTU (up to 512), which allows larger packets and <a href="https://punchthrough.com/maximizing-ble-throughput-part-2-use-larger-att-mtu-2/">slightly</a> higher data throughput. The maximum supported MTU depends on the smartphone’s and Bluetooth device’s chipset, though some device chipsets may not support larger MTUs at all.</p>

<p>Some libraries support changing MTU. Otherwise you can request a larger MTU size with <a href="https://developer.android.com/reference/kotlin/android/bluetooth/BluetoothGatt#requestMtu(kotlin.Int)">BluetoothGatt.requestMtu</a> and the result will be passed to <a href="https://developer.android.com/reference/kotlin/android/bluetooth/BluetoothGattCallback#onMtuChanged(android.bluetooth.BluetoothGatt,%20kotlin.Int,%20kotlin.Int)">BluetoothGattCallback.onMtuChanged</a>.</p>

<p>Using a larger MTU may lead to disconnects on some Android devices (e.g. <a href="https://forum.developer.samsung.com/t/samsung-android-10-ble-connectivity-regression/509">Android 10 Samsung</a>), so only use this if necessary.</p>

<h2 id="bluetooth-le-phy">Bluetooth LE PHY</h2>
<p>On API ≥ 26 the PHY (physical layer) <a href="https://www.bluetooth.com/blog/what-bluetooth-developers-should-know-about-android-o/">Bluetooth 5.0 extensions</a> can be used to control the connection’s <a href="https://en.wikipedia.org/wiki/Bluetooth_Low_Energy#2M_PHY">error correction redundancy and phase modulations</a> which affects bandwidth and range:</p>
<ul>
  <li><a href="https://developer.android.com/reference/kotlin/android/bluetooth/BluetoothDevice#phy_le_1m">PHY_LE_1M</a>: 1 Mb/s and 100% range (default)</li>
  <li><a href="https://developer.android.com/reference/kotlin/android/bluetooth/BluetoothDevice#phy_le_2m">PHY_LE_2M</a>: 2 Mb/s and ~80% range</li>
  <li><a href="https://developer.android.com/reference/kotlin/android/bluetooth/BluetoothDevice#phy_le_coded">PHY_LE_CODED</a>:
    <ul>
      <li><a href="https://developer.android.com/reference/kotlin/android/bluetooth/BluetoothDevice#phy_option_s2">PHY_OPTION_S2</a>: 500 kb/s, 200% range</li>
      <li><a href="https://developer.android.com/reference/kotlin/android/bluetooth/BluetoothDevice#phy_option_s8">PHY_OPTION_S8</a>: 125 kb/s, 400% range</li>
    </ul>
  </li>
</ul>

<p>Setting the PHY (<a href="https://developer.android.com/reference/kotlin/android/bluetooth/BluetoothGatt#setPreferredPhy(kotlin.Int,%20kotlin.Int,%20kotlin.Int)">BluetoothGatt.setPreferredPhy</a>) does not guarantee that it will be used as not all smartphones or Bluetooth devices support all PHY modes.</p>

<h2 id="bluetooth-le-subscriptions">Bluetooth LE Subscriptions</h2>
<p>Instead of repeatedly reading a characteristic to get the current value (polling) you can use Bluetooth LE subscriptions to automatically be informed once the characteristics’s value changes (pushing). There are two types of subscription, which characteristics can support:</p>
<ul>
  <li>Indications: Device requires acknowledgement from smartphone for pushed values (like TCP).</li>
  <li>Notifications: Device does not require acknowledgement (like UDP).</li>
</ul>

<p>The maximum number of open subscriptions <a href="https://stackoverflow.com/questions/42771904/android-bluetooth-low-energy-characteristic-notification-count-limit-does-this">depend on the Android version</a> (more subscriptions are silently ignored):</p>
<ul>
  <li>API ≥ 18 → 3</li>
  <li>API ≥ 19 → 7</li>
  <li>API ≥ 21 → 15</li>
</ul>

<h2 id="google-fast-pair">Google Fast Pair</h2>
<p>Some Bluetooth LE devices support <a href="https://developers.google.com/nearby/fast-pair/help">Google Fast Pair</a>, which is a <a href="https://developers.google.com/nearby/fast-pair/specifications/introduction#gatt_service">special BLE service</a>. When a BLE device enters pairing mode, smartphones in the vicinity will automatically show a notification:</p>

<p><img src="https://developers.google.com/nearby/fast-pair/images/initial-pairing.png" alt="Fast Pair Notification" /></p>

<p>After pairing, Fast Pair may suggest installing the companion app, if the app id is correctly configured in the device’s <a href="https://developers.google.com/nearby">Google Nearby</a> console.</p>

<div class="message">

  <p>I’ve met an issue in the past, where it was not possible to use protected BLE characteristics, when the <a href="https://developer.android.com/reference/android/bluetooth/BluetoothDevice">BluetoothDevice</a> (provided by <a href="https://developer.android.com/reference/android/bluetooth/BluetoothAdapter#getBondedDevices()">BluetoothAdapter.getBondedDevices</a>) was bonded via Fast Pair. This was probably just an issue with the device’s firmware, but still worth to check when dealing with Fast Pair.</p>
</div>

<h2 id="dual-mode-devices">Dual Mode devices</h2>
<p>Some devices support both BLE and Bluetooth Classic and appear as two devices (with different names) in the list of bonded Bluetooth devices (two separate <a href="https://developer.android.com/reference/android/bluetooth/BluetoothDevice">BluetoothDevice</a>, not <code class="language-plaintext highlighter-rouge">DEVICE_TYPE_DUAL</code>). For example headsets, which use Bluetooth Classic for audio transfer and BLE for control (noice cancellation, equalizer, firmware update, etc.). When bonding one type, the other may be bonded automatically as well.</p>

<h2 id="companion-device-pairing">Companion device pairing</h2>
<p><a href="https://developer.android.com/guide/topics/connectivity/companion-device-pairing">Companion device pairing</a> is an alternative for scanning, that provides a system dialog for selecting a device. It does not require any location permissions and works with location service disabled, however bluetooth still needs to be enabled. It can be configured with a scan filter. Bonding has to be done manually.</p>

<p>However it is only available on API ≥ 26 and is very limited, as its <a href="https://www.youtube.com/watch?v=F1mvm_bfNgU">use case</a> seems to be very specific to smart watches and background connectivity. The dialog is only shown as soon as at least one matching device is found, so if no device is in range, no dialog is shown. It does not support continuous scanning, which makes it unusable for onboarding flows where a device is not always discoverable, i.e. if the user has to put in pairing mode first. To retry a scan, the dialog has to be closed and opened again by the user.</p>

<h2 id="unstable-service-record-uuids">Unstable service record UUIDs</h2>
<p>A Bluetooth Classic device’s supported service record UUIDs can be accessed with <a href="https://developer.android.com/reference/kotlin/android/bluetooth/BluetoothDevice#getuuids">BluetoothDevice.getUuids</a>. This is a cache which is created during pairing.</p>

<p>In very rare cases some devices have unstable service record UUIDs (e.g. UUID depends on firmware version). In that case, the cache can can be refreshed by calling <a href="https://developer.android.com/reference/kotlin/android/bluetooth/BluetoothDevice#fetchUuidsWithSdp()">BluetoothDevice.fetchUuidsWithSdp</a> and waiting for the result in a <code class="language-plaintext highlighter-rouge">ACTION_UUID</code> broadcast receiver. Fetching UUIDs takes up to 5 seconds, so don’t do this for every connection attempt, just as a fallback for devices with unstable service records.</p>

<h2 id="randomized-mac-addresses">Randomized MAC addresses</h2>
<p>Most Bluetooth 4.2 devices use a periodically changing randomized MAC address during scanning. This is a <a href="https://www.bluetooth.com/blog/bluetooth-technology-protecting-your-privacy/">privacy feature</a> to prevent tracking. So a MAC address (<a href="https://developer.android.com/reference/kotlin/android/bluetooth/BluetoothDevice#getaddress">BluetoothDevice.getAddress</a>) from a scanned device should not be used to identify a device.</p>

<p>However once paired, the MAC address is stable and can be persisted, e.g. to remember the last connected device if an app allows connecting to multiple devices of the same kind.</p>

<h2 id="bluetooth-on-ios">Bluetooth on iOS</h2>
<p>The iOS Bluetooth API is functionally very different to Android, which leads to different UX in onboarding and connection flows:</p>
<ul>
  <li>Both have different preconditions and permission models for scanning and connecting.</li>
  <li>iOS can’t access the list of bonded devices or know if a device is not paired anymore.</li>
  <li>iOS can’t pair manually. Pairing is always done by reading from a protected characteristic.</li>
  <li>Bluetooth Classic requires a special <a href="https://mfi.apple.com/">MFI</a> chip on the Bluetooth device and MFI certification of both app and device.</li>
</ul>

<h1 id="libraries">Libraries</h1>
<ul>
  <li><a href="https://github.com/dariuszseweryn/RxAndroidBle">RxAndroidBle</a>: Very stable BLE Library with support for permission handling, scanning, communication, long writes, setting MTU, etc. Uses reactive programming with <a href="https://github.com/ReactiveX/RxJava">RxJava</a>. <strong>Recommendation</strong>: Combine this with <a href="https://github.com/Kotlin/kotlinx.coroutines/tree/master/reactive/kotlinx-coroutines-rx2">kotlinx-coroutines-rx2</a> adapters, so you can use coroutines and flows.</li>
  <li><a href="https://github.com/JuulLabs/kable">Kable</a>: Kotlin-Multi-Platform BLE library (Android, iOS, Web). Supports scanning and communication. Uses Kotlin coroutines and flows.</li>
  <li><a href="https://github.com/IvBaranov/RxBluetooth">RxBluetooth</a>: Bluetooth Classic library with some nice abstractions. Uses reactive programming with <a href="https://github.com/ReactiveX/RxJava">RxJava</a>.</li>
  <li><a href="https://github.com/NordicSemiconductor/Android-BLE-Library">Android-BLE-library</a>: I have not used this yet, but it is made by Nordic Semiconductor who brought us the <a href="https://play.google.com/store/apps/details?id=no.nordicsemi.android.mcp&amp;hl=de&amp;gl=US">nRF Connect</a> app, which is really great for debugging.</li>
</ul>]]></content><author><name>Mike Klimek</name></author><category term="Android" /><summary type="html"><![CDATA[A comprehensive guide to Bluetooth on Android.]]></summary></entry><entry><title type="html">Problem Decomposition</title><link href="https://mike.cloud/software%20design/2021/01/26/problem-decomposition.html" rel="alternate" type="text/html" title="Problem Decomposition" /><published>2021-01-26T00:00:00+00:00</published><updated>2021-01-26T00:00:00+00:00</updated><id>https://mike.cloud/software%20design/2021/01/26/problem-decomposition</id><content type="html" xml:base="https://mike.cloud/software%20design/2021/01/26/problem-decomposition.html"><![CDATA[<p>Software design is all about <a href="https://en.wikipedia.org/wiki/Decomposition_(computer_science)">problem decomposition</a>. Take a large problem, break it up into smaller more managable problems, and you can solve each one individually. This allows you to iteratively solve every problem, no matter how large, in small steps.</p>

<p>Problem decomposition is what we do when we:</p>
<ul>
  <li>Design software architecture (MVVM, layers, etc.)</li>
  <li>Structure code units (modules, packages, classes, functions)</li>
  <li>Design algorithms (“divide and conquer”)</li>
  <li>Split up tasks (features, epics, etc.)</li>
  <li>Design UX and UI (components, layouts, user flows, etc.)</li>
  <li>And much more…</li>
</ul>

<h1 id="how-to-decompose-a-problem">How to decompose a problem?</h1>

<p>To decompose a problem, we have to answers these questions:</p>
<ul>
  <li>What are the “breaking points” for splitting something up?</li>
  <li>When is a problem space small enough, so we can stop breaking it up?</li>
</ul>

<p>As with nearly everything in software design there are no <em>one size fits all</em> answers. The <a href="/software%20design/2020/12/17/principles.html">design principles</a> that help anwers these questions are these:</p>
<ul>
  <li><a href="https://en.wikipedia.org/wiki/Cohesion_(computer_science)">High Cohesion</a>: Put stuff together that belongs together. E.g. put all user management code into a <code class="language-plaintext highlighter-rouge">user</code> directory, instead of generic <code class="language-plaintext highlighter-rouge">viewmodel</code>, <code class="language-plaintext highlighter-rouge">ci</code>, <code class="language-plaintext highlighter-rouge">model</code> directories that are also used for other features (<a href="http://www.javapractices.com/topic/TopicAction.do?Id=205">package by feature, not type</a>). <strong>The structure is not right, if you have to jump a lot between files and directories while developing or maintaining a feature.</strong></li>
  <li><a href="https://en.wikipedia.org/wiki/Loose_coupling">Loose Coupling</a>: Try to reduce the number of possible dependencies (a.k.a spaghetti) between modules to a minimum. E.g. reduce the number of exposed (<code class="language-plaintext highlighter-rouge">public</code>) methods and classes. If you think something could be used in another module later, assume YAGNI. You can still expose it later or refactor it to a shared module. <strong>The structure is right, if you can just rip out or refactor a module, and don’t need to extensively rewrite other modules.</strong></li>
  <li>Hierarchy: When decomposing problems and subproblems, you will produce a tree structure. This should be abstract and domain specific at the top (e.g. screen for coffee maker app) and become more generic and detailled towards the bottom (text field components, generic data manipulation algorithms). <strong>The structure is right, if it enables extensive domain specific changes, without having to rewrite implementation details.</strong></li>
</ul>

<p><img src="https://upload.wikimedia.org/wikipedia/commons/thumb/0/09/CouplingVsCohesion.svg/1200px-CouplingVsCohesion.svg.png" alt="Cohesion" /></p>

<h1 id="example-stepwise-refinment">Example: Stepwise Refinment</h1>
<p>Stepwise refinement is a top down development technique for tackling a problem by dividing it into multiple smaller abstract steps. More abstract code is written first (top) and implementation details later (down), comparable to a breadth first search.</p>

<p>This is especially useful for functional reactive programming, because every step can be build and previewed.</p>

<ol>
  <li>
    <p>Define method with an empty body (“stub”). The method name is an abstraction of what you want to achieve.</p>

    <div class="language-kotlin highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="k">fun</span> <span class="nf">renderScreen</span><span class="p">()</span> <span class="p">{</span>
 <span class="p">}</span>
</code></pre></div>    </div>
  </li>
  <li>
    <p>Fill in the method’s body with methods which are not defined yet:</p>

    <div class="language-kotlin highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="k">fun</span> <span class="nf">renderScreen</span><span class="p">()</span> <span class="p">=</span> <span class="nc">Column</span> <span class="p">{</span>
     <span class="nf">renderHeader</span><span class="p">()</span>
     <span class="nf">renderBody</span><span class="p">()</span>
 <span class="p">}</span>
</code></pre></div>    </div>
  </li>
  <li>
    <p>Define new methods:</p>

    <div class="language-kotlin highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="k">fun</span> <span class="nf">renderScreen</span><span class="p">()</span> <span class="p">=</span> <span class="nc">Column</span> <span class="p">{</span>
     <span class="nf">renderHeader</span><span class="p">()</span>
     <span class="nf">renderBody</span><span class="p">()</span>
 <span class="p">}</span>

 <span class="k">fun</span> <span class="nf">renderHeader</span><span class="p">()</span> <span class="p">{}</span>
 <span class="k">fun</span> <span class="nf">renderBody</span><span class="p">()</span> <span class="p">{}</span>
</code></pre></div>    </div>
  </li>
  <li>
    <p>Fill in method bodies:</p>

    <div class="language-kotlin highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="k">fun</span> <span class="nf">renderScreen</span><span class="p">()</span> <span class="p">=</span> <span class="nc">Column</span> <span class="p">{</span>
     <span class="nf">renderHeader</span><span class="p">()</span>
     <span class="nf">renderBody</span><span class="p">()</span>
 <span class="p">}</span>

 <span class="k">fun</span> <span class="nf">renderHeader</span><span class="p">()</span> <span class="p">=</span> <span class="nc">Row</span> <span class="p">{</span>
     <span class="nf">renderIcon</span><span class="p">()</span>
     <span class="nf">renderTitle</span><span class="p">()</span>
 <span class="p">}</span>

 <span class="k">fun</span> <span class="nf">renderBody</span><span class="p">()</span> <span class="p">=</span> <span class="nf">renderList</span><span class="p">()</span>
</code></pre></div>    </div>
  </li>
  <li>
    <p>Etc.</p>

    <div class="language-kotlin highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="k">fun</span> <span class="nf">renderScreen</span><span class="p">()</span> <span class="p">=</span> <span class="nc">Column</span> <span class="p">{</span>
     <span class="nf">renderHeader</span><span class="p">()</span>
     <span class="nf">renderBody</span><span class="p">()</span>
 <span class="p">}</span>

 <span class="k">fun</span> <span class="nf">renderHeader</span><span class="p">()</span> <span class="p">=</span> <span class="nc">Row</span> <span class="p">{</span>
     <span class="nf">renderIcon</span><span class="p">()</span>
     <span class="nf">renderTitle</span><span class="p">()</span>
 <span class="p">}</span>

 <span class="k">fun</span> <span class="nf">renderBody</span><span class="p">()</span> <span class="p">=</span> <span class="nf">renderList</span><span class="p">()</span>
 <span class="k">fun</span> <span class="nf">renderList</span><span class="p">()</span> <span class="p">=</span> <span class="nf">listOf</span><span class="p">(</span><span class="s">"Foo"</span><span class="p">,</span> <span class="s">"Bar"</span><span class="p">).</span><span class="nf">map</span><span class="p">(</span><span class="n">renderItem</span><span class="p">)</span>

 <span class="k">fun</span> <span class="nf">renderIcon</span><span class="p">()</span> <span class="p">=</span> <span class="nc">Icon</span><span class="p">(</span><span class="nc">Icons</span><span class="p">.</span><span class="n">logo</span><span class="p">)</span>
 <span class="k">fun</span> <span class="nf">renderTitle</span><span class="p">()</span> <span class="p">=</span> <span class="nc">Text</span><span class="p">(</span><span class="s">"Hello World"</span><span class="p">)</span>
 <span class="k">fun</span> <span class="nf">renderItem</span><span class="p">(</span><span class="n">item</span><span class="p">)</span> <span class="p">=</span> <span class="nc">Text</span><span class="p">(</span><span class="n">item</span><span class="p">)</span>
</code></pre></div>    </div>
  </li>
</ol>]]></content><author><name>Mike Klimek</name></author><category term="Software Design" /><summary type="html"><![CDATA[Problem decomposition will allow you to tackle any problem.]]></summary></entry><entry><title type="html">Design Principles</title><link href="https://mike.cloud/software%20design/2020/12/17/principles.html" rel="alternate" type="text/html" title="Design Principles" /><published>2020-12-17T00:00:00+00:00</published><updated>2020-12-17T00:00:00+00:00</updated><id>https://mike.cloud/software%20design/2020/12/17/principles</id><content type="html" xml:base="https://mike.cloud/software%20design/2020/12/17/principles.html"><![CDATA[<p>These principles are open to interpretation and not strict rules to blindly follow. There is no correct way to write clean code, as it requires a lot of creativity, but you can use these principles as inspiration:</p>

<details>
  <summary>Keep It Simple and Stupid (KISS)</summary>

  <ul>
    <li>Less is more. If something is complex it will likely break. So complete a feature with minimum amount of necessary code, that is still perfectly understandable. Try to find the “cleanest” solution.</li>
    <li>Development flow: Make it work, then iteratively refactor to make it more simple/readable. Maybe there is a function in the standard library you can use instead, or you can simplify a layout, or some duplicate code can be extracted. Before submitting a merge request, check if you can simplify your changes even further.</li>
  </ul>

  <p>This reduces the overall error surface (less code → less potential bugs) and reduces the maintainable surface (less code to touch when refactoring → more flexible).</p>
</details>

<details>
  <summary>Single Responsibility</summary>

  <ul>
    <li>Methods should only do one thing (e.g. display, calculate, create, combine, save, get, set, enable, disable, fetch, query, show, hide, etc.) which is reflected in the method name. If there is an “and” in the name, you can probably split it.</li>
    <li>Rule of thumb: Methods should be roughly less than ~20 lines and fit on page without scrolling. If it is too long, then this is an indicator, that it is doing too much and should be split up. <strong>This is of course only a rough guideline and not always applicable, so don’t enforce it!</strong></li>
    <li>Methods should be roughly on the same layer of abstraction. E.g. if you have function <code class="language-plaintext highlighter-rouge">renderList</code>, then put the code to render items into a separate <code class="language-plaintext highlighter-rouge">renderItem</code> function, instead of adding this code to the <code class="language-plaintext highlighter-rouge">renderList</code> function.</li>
    <li>Classes should only be responsible for a single feature. E.g. try to prevent an abstract base class, where every shared code is dumped, instead pull this out into extension functions or components.</li>
    <li>Don’t mix business logic details (e.g. SQL database queries) with presentation details (direct view manipulation, animations), instead use a mediator (controller, viewmodel, component, etc.) between classes that manipulate views and classes that handle data sources.</li>
  </ul>
</details>

<details>
  <summary>Loose Coupling</summary>

  <ul>
    <li>Relevant principles: Information Hiding, Separation of Concerns, Demeter Law, Interface Segregation, Dependency Inversion</li>
    <li>Reduce scope (references to other classes, member variables, parameters, etc.), e.g. by hiding implementation details behind private fields and methods or using pure functions. <strong>Pure functions</strong> have no side effects or state (no reference to members) and always give the same output for the same input (testable).</li>
    <li>A classes’ or function’s dependencies should be obvious, when looking at its signature (function parameters, constructor parameters). <strong>Never use global state and singletons</strong>, as this will hide the fact that a class or function depends on another component, without looking at its implementation. Global state will make it really hard to follow the code, as you have to keep in mind every single place it may be modified and where it is used. Global state also makes code nearly untestable, as dependencies can’t be replaced easily.</li>
    <li>Parameters should not contain more information than a function or class actually needs I.e. don’t pass around a massive context object or the whole viewmodel, if just a callback would suffice.</li>
  </ul>

  <p>Loose coupling makes it easier to create a mental model and follow the flow of the code. The code is easier to test and verify, when there are less influencing external factors and we can be more confident, that the code is correct.</p>
</details>

<details>
  <summary>Principle of least Surprise (POLS)</summary>

  <ul>
    <li>A unit’s (methods, class) name is a mental abstraction for its functionality. So reading the name (or signature) should be enough information to use it, without unexpected behavior. When encountering unexpected behavior, we have to look at the implementation details, which consumes time and reduces our trust in the code. Good code looks <strong>boring</strong> as it contains no such surprises.</li>
    <li>If something is not obvious (<a href="https://en.wikipedia.org/wiki/Magic_(programming)">“black magic”</a>, workarounds, bugs in frameworks, deviation from best practices, etc.) or may lead to issues in the future, mark it as a place that needs special attention. E.g. by adding a comment or pulling it into a separate function, where the method names describes the behavior. This prevents surprises, communication overhead and regression, when other developers touch this piece of code.</li>
    <li>Prefer appropriately named methods to comments (<strong>self documenting code</strong>, i.e. code is the ground truth) and don’t add comments to obvious code. Comments increase the maintenance surface and can be misleading (compiler can’t check comments for correctness, comments are usually never updated). Comments reduce readability (code does not fit on single page, reading flow is broken).</li>
  </ul>
</details>

<details>
  <summary>Don't repeat yourself (DRY)</summary>

  <ul>
    <li>Ritualistic copy-pasting code is easy, but it is likely a sign of <a href="https://en.wikipedia.org/wiki/Cargo_cult_programming">cargo cult programming</a>, i.e. not understanding the reason behind the code. Make sure you know how the code works, maybe there is a better way to do this.</li>
    <li>Copy-pasting code very often leads to copy-paste errors (e.g. something is not renamed). Once there is a change/refactoring to the copied code, the pasted code usually needs to be manually updated in all place (no single source of truth, large maintenance surface), which is easily forgotten. This frequently leads to bugs, that you thought were already fixed.</li>
    <li>Instead move duplicated logic to helper functions, e.g. an extension function for creating a toast message, so the core logic is in a single place (single source of truth) and can be easily changed.</li>
    <li>In general try to keep the signal to noise (business logic code vs. glue code) ration high.</li>
    <li>But: Don’t over-engineer, which may obstruct further development and confuse new developers. Keep it easy to adapt and extend. Sometimes it is better to be flexible, e.g. copy an &lt;ImageView&gt; instead of moving it to a shared &lt;include&gt; layout, if you know the design for it will be different every time.</li>
  </ul>
</details>

<details>
  <summary>You aren't gonna need it (YAGNI)</summary>

  <ul>
    <li><strong>No dead or commented out code.</strong> This increases maintenance surface or even worse is not maintained at all (not type checked by the compiler). If this is really needed, then only with an explanatory comment why this is still there or when it can be removed, e.g. if dependent on a feature that is developed in parallel. Otherwise this will be forever in the codebase, and other developers will not know if this is still needed or can be safely removed.</li>
    <li>Don’t wrap every class in an explicit interface class. Classes already provide an interface via their public properties and methods (loose coupling). Interfaces reduce development velocity, because now there are two places (interface and implementation) that need to be changed when adding functionality. Interfaces make it hard to follow the code without actually running/debugging the code. Usually it is easy to add an interface later once it is needed, e.g. if there is actually more than one implementation. However for libraries or public APIs an interface adds an additional layer of separation.</li>
    <li>Don’t over-engineer and keep your code flexible.</li>
    <li>But: Some things are not initially planned, but are guaranteed to be needed later in nearly ever project, like error handling or localization.</li>
  </ul>
</details>

<details>
  <summary>Composition over Inheritance</summary>

  <ul>
    <li>Don’t use abstract “base” classes, which grow huge and can’t be split up. They also don’t work with multiple inheritance, e.g. when a framework provides its own base class we have to use. Instead put functionality in pure (extension) functions, or delegate it to separate components, which can be injected where needed.</li>
    <li>Compose your data structures out of other data structures. This simplifies integration with other formats that do not have inheritance, like JSON or relational databases.</li>
  </ul>
</details>

<details>
  <summary>Convention over Configuration</summary>

  <ul>
    <li><strong>Follow the best practices</strong> and don’t fight the system. Usually there is a cleaner and better solution in the official documentation than in StackOverflow code snippets. E.g. for Android it is often suggested to use a custom background to change a button’s color, but this breaks animations unlike the best practice (theming).</li>
    <li>Use the recommended code style for the project’s platform and enforce it via CI and code reviews (even if it seems pedantic). This makes the code more recognizable, it is easier to read and change other developer’s code, and there are less conflicts when autoformatting code.</li>
    <li>A developer should be able to <strong>build a project immediately after a checkout</strong> with as little effort or help as possible. This should require no complex build process (KISS) and should follow platform convention, e.g. <code class="language-plaintext highlighter-rouge">gradle assembleDebug</code> for Android or <code class="language-plaintext highlighter-rouge">docker-compose up</code> for backends. Defaults should be provided if possible, so there is no need to configure urls, accounts, keys, etc. If a defaults can not be provided, consider disabling a feature, e.g. use a debug key for signing or providing a template config file, which can be filled in.</li>
    <li>Deployment should be as easy as possible (KISS) for every developer via CI/CD and should only take a few minutes, e.g. merging <code class="language-plaintext highlighter-rouge">develop</code> on <code class="language-plaintext highlighter-rouge">master</code>. Deployment should not be done locally (e.g. deploying a partial development state).</li>
    <li>Keep the CI/CD config simple (KISS) but effective, so there is less time needed for configuring or debugging the CI and more time for coding. If something is too complex it will likely break, so simplify it or make it easy for a developer to do manually. CI should include at least an automatic build (smoke test). Better yet a code quality check (linter), running tests (even if there are no tests written yet) and deployment.</li>
    <li>Technical documentation (README.md) should provide some of the following:
      <ul>
        <li>What is this project/codebase about?</li>
        <li>Special requirements / tools</li>
        <li>How to build</li>
        <li>How to deploy / create release</li>
        <li>Some helpful guides / commands (e.g. architecture/structure, how to update translations, how to make migration, how to fix common issues, URLs for services, etc.)</li>
      </ul>
    </li>
  </ul>
</details>

<details>
  <summary>Test-driven development (TDD)</summary>

  <ul>
    <li>TDD is a development technique that also produces a test suite that has high test coverage and protects against future regressions, however TDD is not universally applicable to every task.</li>
    <li>Basic development workflow:
      <ol>
        <li>Write test for a tiny part of new feature</li>
        <li>Write code until test works</li>
        <li>Refactor / clean up code</li>
        <li>Make sure tests still works (no regression)</li>
        <li>Repeat until feature is done</li>
      </ol>
    </li>
    <li>Only one thing should be tested per test. For this you can structure tests into three parts, which also helps other developers understand and modify your tests:
      <ol>
        <li><strong>Given</strong>: Setup the precondition state, e.g. create the class you want to test and write some  dummy data into it’s fields.</li>
        <li><strong>When</strong>: The action you want to test, e.g. call a method on your class.</li>
        <li><strong>Then</strong>: The expected postcondition, e.g. certain fields in your class should have new values.</li>
      </ol>
    </li>
    <li>What should (not) be tested?
      <ul>
        <li>No 100% coverage: Tests increase the maintainable surface of the codebase and reduce development velocity, as they need to be rewritten, when a feature changes or is refactored. So the goal should be not to test everything, but to reduce complexity, so <strong>less tests are actually needed</strong>.</li>
        <li>Most code units should be clean and self evident with high confidence, so tests would be redundant. Test only where it makes sense. E.g. don’t test getters/setter or a function that contains only a simple if/else control flow.</li>
        <li>Good candidates for testing and TDD are complex pure functions (e.g. parsers or date time calculation), code that needs high confidence (e.g. payment processing), or code that can not be easily tested manually (rare crash conditions and edge cases).</li>
        <li>Don’t test implementation details of external dependencies, e.g. if a parser generated by a code generator can parse every datatype correctly. For critical dependencies a boundary or API test should suffice, so you know early on if it is safe to upgrade the library to a new version.</li>
        <li>Tests may also contain bugs, so a test should usually not be more complex than the code it tests.</li>
        <li>UI tests (e.g. on an emulator or browser) are slow and fragile, so prefer unit tests. Viewmodels should act as <a href="https://martinfowler.com/bliki/HumbleObject.html">humble objects</a>.</li>
        <li>Tests can’t find high level gaps in specification or overall behavior, so explorative QA testing is still needed.</li>
      </ul>
    </li>
    <li>If projects don’t have tests (or support for tests) from the start, they likely won’t have tests later. Usually there is considerable effort involved in starting to write tests, e.g. mocking dependencies, separating framework from test code, etc. To allow easy testing without obstructions or effort there should be at least one moderately complex test case when starting a new project, that can be used as a template, even if there are no other tests planned yet.</li>
  </ul>
</details>

<details>
  <summary>Graceful Degradation</summary>

  <ul>
    <li>There are two kinds of errors:
      <ul>
        <li>If an error would inevitably leave your program in an illegal or broken state that would cause other issues (e.g. data corruption), then it is <strong>fatal</strong> and should crash the program.</li>
        <li>If you can recover from the error, so your program always remains in a valid state, then it is an <strong>exception</strong> and you should handle it. E.g. you could show a message to your users and let them try again after a failed backend request. Your program’s state must always stay valid, e.g. when using optimistic request (showing result before a request is finished), then the previous state (before result was shown) must be restored.</li>
      </ul>
    </li>
    <li><strong>Never silently suppress exceptions</strong> (empty catch block). Swallowed exceptions make it very hard to find the source of bugs and may introduce bugs that could easily be discovered early during development. Instead at least log it and/or pass it to the user as a generic error (in a server return status 500). In rare cases where a specific type of exception is expected and can safely be ignored, add a comment, so it is clear why there is no logging.</li>
  </ul>
</details>]]></content><author><name>Mike Klimek</name></author><category term="Software Design" /><summary type="html"><![CDATA[A concise overview of software design principles.]]></summary></entry></feed>